«

Hi,

Can anyone give the best way (faster) to write the vector contents to a file?

Is there a better way to do it than below method?

for(unsigned i=0;i<terms.size();i++){
sprintf(temp,"%d %s\n",i,terms[i].c_str());
ids_n_terms = ids_n_terms + temp;
}

Is the string addition a very expensive operation? (will a new memory allocation takes place)

there are around 5,00,000 (strings) in the vector, and these have to stored in
<id term> format.

Thanks,
Sravan.

On 09/20/11 03:05 PM, sravanreddy001 wrote:
> Hi,
>
> Can anyone give the best way (faster) to write the vector contents to a file?
>
> Is there a better way to do it than below method?
>
> for(unsigned i=0;i<terms.size();i++){
> sprintf(temp,"%d %s\n",i,terms[i].c_str());
> ids_n_terms = ids_n_terms + temp;
> }

Where do you do the write?

> Is the string addition a very expensive operation? (will a new memory allocation takes place)
>
> there are around 5,00,000 (strings) in the vector, and these have to stored in
> <id term> format.

You will probably be I/O limited, so optimising your code won't make any
significant difference.

--
Ian Collins

I'm actually writing this to a file.
But, file operation is just one simple write.
(the string that is constructed in the loop is written.. and the file size is around 5 MB.)

but.. string+ = string + some_thing;

if this is going to allocate a new memory location, then.. writing it multiple times to file.. will any of this approach help?

One my i3 2nd gen computer.. it took around 10 mins. (using one single core)
not multithreaded.

Is there a way to automatically push the vector contents to file in an efficient manner?

On 09/20/11 03:28 PM, sravanreddy001 wrote:

Context is nice!

> I'm actually writing this to a file.
> But, file operation is just one simple write.
> (the string that is constructed in the loop is written.. and the file size is around 5 MB.)
>
> but.. string+ = string + some_thing;
>
> if this is going to allocate a new memory location, then.. writing it multiple times to file.. will any of this approach help?
>
> One my i3 2nd gen computer.. it took around 10 mins. (using one single core)
> not multithreaded.

You would probably be better off just using an ofstream and writing a
line at a time. I have one application that reads and writes in excess
of 1GB with fstreams. Reading takes a couple of minutes, writing about 5.

Your huge string will do a lot of progressively bigger memory moves,
which are a bigger issue than the allocations. Kind of a
macro-deoptimisation!

--
Ian Collins

do you think, writing so many times... 5 lakhs times(lines) is more expencive... more disk writes.. thats why i came to that approach..

so.. will wiriting to file every 20-30 lines at once.. will it help me?
it will atleast reduce the disk writes by 20-30 times..

Also, I'm reading all the file content in one GO.

seeking until the end of file.. read from start to end.
this is efficient than reading line by line right?

On 09/20/11 03:50 PM, sravanreddy001 wrote:
> Also, I'm reading all the file content in one GO.

Please keep the context from the post you are relying to, otherwise your
posts don't make sense.

> seeking until the end of file.. read from start to end.
> this is efficient than reading line by line right?

You are worrying about the wrong thing, just create an ifstream and read
the entries in one at a time.

--
Ian Collins

On 09/20/11 03:47 PM, sravanreddy001 wrote:
> do you think, writing so many times... 5 lakhs times(lines) is more expencive... more disk writes.. thats why i came to that approach..
>
> so.. will wiriting to file every 20-30 lines at once.. will it help me?
> it will atleast reduce the disk writes by 20-30 times..

Leave the buffering to the stream and the OS, it's their job. Unless
you flush the stream each line and force synchronous writes on the file,
you will not see millions of writes. On my system, that much data would
be coalesced into one disk write!

--
Ian Collins

sravanreddy001 <> wrote:
> for(unsigned i=0;i<terms.size();i++){
> sprintf(temp,"%d %s\n",i,terms[i].c_str());
> ids_n_terms = ids_n_terms + temp;
> }

I don't understand why you can't write to the file directly. Creating
the contents of the file into a dynamic string first like that is
significantly more inefficient.