Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > allocators and string

Reply
Thread Tools

allocators and string

 
 
Christopher
Guest
Posts: n/a
 
      12-13-2011
Admittedly, I've never had to worry about making a custom allocator. I
think that is what I need. I need to replace a broken buffer, where it
looks like the only real goal was to allocate large chunks of storage
at a time instead of every insertion.

Where do I start?

It isn't standard or documented how much a string allocates when an
insertion causes it to run out of space, is it?
I've heard things like "double the current."

If I want to the string to allocate space for 50,000 elements at a
time, every time? How do I go about that.

 
Reply With Quote
 
 
 
 
Christopher
Guest
Posts: n/a
 
      12-14-2011
On Dec 13, 5:08*pm, Paavo Helde <(E-Mail Removed)> wrote:

>
> I think STL allocators do not decide how much to allocate, they just
> allocate.
>
> You can wrap std::string in a custom class and use the capacity() and
> reserve() member functions for checking and enlarging the buffer.
> However, from your description it sounds like std::string might not be
> the best fit here. What problem are you trying to resolve?



Well the broken implementation that I am trying to replace is just a
buffer to grab XML from an http response. The author evidently tried
to make it faster by allocating configurable sized chunks of chars and
adding them to a linked list as they come in. Then allocating one
after calculating the total size of all and copying on when
extracting. It doesn't seem to make much of a difference and throws
exceptions which I've given up trying to fix. I read something similar
called "simple segregation" when looking up allocators.

The things I know of that would effect design are
* I get variable sized char data from the http library.
* I get the data in multiple calls and need to append often
* The total size of the data tends to be very large ~50,000 characters
worth of xml document on average, but can be tiny up to a gig of data.
* I need to give the caller whom wishes to extract the entire contents
in contiguous memory at once
 
Reply With Quote
 
 
 
 
Christopher
Guest
Posts: n/a
 
      12-14-2011
P.S. I could just use std::wstring as is, but if speed by eliminating
allocations is possible, it would be a good time for me to learn how
to do it instead of always depending on default STL behavior.

 
Reply With Quote
 
Goran
Guest
Posts: n/a
 
      12-14-2011
On Dec 14, 8:40*am, Christopher <(E-Mail Removed)> wrote:
> On Dec 13, 5:08*pm, Paavo Helde <(E-Mail Removed)> wrote:
>
>
>
> > I think STL allocators do not decide how much to allocate, they just
> > allocate.

>
> > You can wrap std::string in a custom class and use the capacity() and
> > reserve() member functions for checking and enlarging the buffer.
> > However, from your description it sounds like std::string might not be
> > the best fit here. What problem are you trying to resolve?

>
> Well the broken implementation that I am trying to replace is just a
> buffer to grab XML from an http response. The author evidently tried
> to make it faster by allocating configurable sized chunks of chars and
> adding them to a linked list as they come in. Then allocating one
> after calculating the total size of all and copying on when
> extracting. It doesn't seem to make much of a difference and throws
> exceptions which I've given up trying to fix. I read something similar
> called "simple segregation" when looking up allocators.
>
> The things I know of that would effect design are
> * I get variable sized char data from the http library.
> * I get the data in multiple calls and need to append often
> * The total size of the data tends to be very large ~50,000 characters
> worth of xml document on average, but can be tiny up to a gig of data.
> * I need to give the caller whom wishes to extract the entire contents
> in contiguous memory at once


That's a shame. Normally, XML parsers know how to receive some sort of
a stream and work off that. Streams get rid of all your
considerations.

That said, if you know approx. max size, perhaps, as Paavo said, you
should play with reserve(). std::vector might be a better fit, too.

Goran.
 
Reply With Quote
 
Christopher
Guest
Posts: n/a
 
      12-14-2011
On Dec 14, 12:43*pm, Paavo Helde <(E-Mail Removed)> wrote:
> Christopher <(E-Mail Removed)> wrote in news:f20c0d81-495c-4ffc-88da-
> (E-Mail Removed):
>
> > The things I know of that would effect design are
> > * I get variable sized char data from the http library.
> > * I get the data in multiple calls and need to append often
> > * The total size of the data tends to be very large ~50,000 characters
> > worth of xml document on average, but can be tiny up to a gig of data.

>
> Wow, 1G XML files! I hope you have 64-bit compilations.


Yea, tell me about it. I've never seen a project where anything and
everything is represented as xml rather than having a domain and
classes for representing business objects. It is out of my hands. It's
ironic that there is an issue with speed and a lesser issue with
memory usage.

> I assume that the size of the packet is not known in advance, otherwise
> it would be easy to allocate the right buffer in the beginning.


Correct.

> So what was the problem again? Are you concerned about speed, memory
> usage or something else?


Speed primarily. Memory also, but not as much as speed.

> Have you actually verified that standard
> std::string append functionality does not meet your needs? The
> exponential growth of the buffer is specifically designed to give
> amortized linear complexity in such usage scenarios, without knowing the
> required buffer size in advance it would be hard to compete with that.


Well, who is to say that std::string doesn't suit my needs until there
is something to compare it against. It is more of an investigation
into whether or not a better speed could be achieved. The original
author of the broken buffer claimed to achieve better speed, but the
performace tests are skewed and incorrect anyway.





 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
a few stl container questions: allocators and comparers. sheam C++ 7 06-12-2008 10:06 AM
do you know good books/sites about creating custom allocators, streams and iterators? Diego Martins C++ 11 08-29-2007 12:52 AM
Mixing allocators and unitialized_fill? Jef Driesen C++ 1 08-29-2006 02:03 PM
std containers and allocators. Andy Venikov C++ 4 07-13-2005 01:30 PM
Re: allocators...exceptions...new/delete Ross Ridge C++ 2 07-07-2003 08:07 AM



Advertisments