Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > Best way to write the vector content to a file.

Reply
Thread Tools

Best way to write the vector content to a file.

 
 
sravanreddy001
Guest
Posts: n/a
 
      09-20-2011
Hi,

Can anyone give the best way (faster) to write the vector contents to a file?

Is there a better way to do it than below method?

for(unsigned i=0;i<terms.size();i++){
sprintf(temp,"%d %s\n",i,terms[i].c_str());
ids_n_terms = ids_n_terms + temp;
}

Is the string addition a very expensive operation? (will a new memory allocation takes place)

there are around 5,00,000 (strings) in the vector, and these have to stored in
<id term> format.

Thanks,
Sravan.
 
Reply With Quote
 
 
 
 
Ian Collins
Guest
Posts: n/a
 
      09-20-2011
On 09/20/11 03:05 PM, sravanreddy001 wrote:
> Hi,
>
> Can anyone give the best way (faster) to write the vector contents to a file?
>
> Is there a better way to do it than below method?
>
> for(unsigned i=0;i<terms.size();i++){
> sprintf(temp,"%d %s\n",i,terms[i].c_str());
> ids_n_terms = ids_n_terms + temp;
> }


Where do you do the write?

> Is the string addition a very expensive operation? (will a new memory allocation takes place)
>
> there are around 5,00,000 (strings) in the vector, and these have to stored in
> <id term> format.


You will probably be I/O limited, so optimising your code won't make any
significant difference.

--
Ian Collins
 
Reply With Quote
 
 
 
 
sravanreddy001
Guest
Posts: n/a
 
      09-20-2011
I'm actually writing this to a file.
But, file operation is just one simple write.
(the string that is constructed in the loop is written.. and the file size is around 5 MB.)

but.. string+ = string + some_thing;

if this is going to allocate a new memory location, then.. writing it multiple times to file.. will any of this approach help?

One my i3 2nd gen computer.. it took around 10 mins. (using one single core)
not multithreaded.
 
Reply With Quote
 
sravanreddy001
Guest
Posts: n/a
 
      09-20-2011
Is there a way to automatically push the vector contents to file in an efficient manner?
 
Reply With Quote
 
Ian Collins
Guest
Posts: n/a
 
      09-20-2011
On 09/20/11 03:28 PM, sravanreddy001 wrote:

Context is nice!

> I'm actually writing this to a file.
> But, file operation is just one simple write.
> (the string that is constructed in the loop is written.. and the file size is around 5 MB.)
>
> but.. string+ = string + some_thing;
>
> if this is going to allocate a new memory location, then.. writing it multiple times to file.. will any of this approach help?
>
> One my i3 2nd gen computer.. it took around 10 mins. (using one single core)
> not multithreaded.


You would probably be better off just using an ofstream and writing a
line at a time. I have one application that reads and writes in excess
of 1GB with fstreams. Reading takes a couple of minutes, writing about 5.

Your huge string will do a lot of progressively bigger memory moves,
which are a bigger issue than the allocations. Kind of a
macro-deoptimisation!

--
Ian Collins
 
Reply With Quote
 
sravanreddy001
Guest
Posts: n/a
 
      09-20-2011
do you think, writing so many times... 5 lakhs times(lines) is more expencive... more disk writes.. thats why i came to that approach..

so.. will wiriting to file every 20-30 lines at once.. will it help me?
it will atleast reduce the disk writes by 20-30 times..
 
Reply With Quote
 
sravanreddy001
Guest
Posts: n/a
 
      09-20-2011
Also, I'm reading all the file content in one GO.

seeking until the end of file.. read from start to end.
this is efficient than reading line by line right?
 
Reply With Quote
 
Ian Collins
Guest
Posts: n/a
 
      09-20-2011
On 09/20/11 03:50 PM, sravanreddy001 wrote:
> Also, I'm reading all the file content in one GO.


Please keep the context from the post you are relying to, otherwise your
posts don't make sense.

> seeking until the end of file.. read from start to end.
> this is efficient than reading line by line right?


You are worrying about the wrong thing, just create an ifstream and read
the entries in one at a time.

--
Ian Collins
 
Reply With Quote
 
Ian Collins
Guest
Posts: n/a
 
      09-20-2011
On 09/20/11 03:47 PM, sravanreddy001 wrote:
> do you think, writing so many times... 5 lakhs times(lines) is more expencive... more disk writes.. thats why i came to that approach..
>
> so.. will wiriting to file every 20-30 lines at once.. will it help me?
> it will atleast reduce the disk writes by 20-30 times..


Leave the buffering to the stream and the OS, it's their job. Unless
you flush the stream each line and force synchronous writes on the file,
you will not see millions of writes. On my system, that much data would
be coalesced into one disk write!

--
Ian Collins
 
Reply With Quote
 
Juha Nieminen
Guest
Posts: n/a
 
      09-20-2011
sravanreddy001 <(E-Mail Removed)> wrote:
> for(unsigned i=0;i<terms.size();i++){
> sprintf(temp,"%d %s\n",i,terms[i].c_str());
> ids_n_terms = ids_n_terms + temp;
> }


I don't understand why you can't write to the file directly. Creating
the contents of the file into a dynamic string first like that is
significantly more inefficient.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
const vector<A> vs vector<const A> vs const vector<const A> Javier C++ 2 09-04-2007 08:46 PM
Initializing vector<vector<int> > and other vector questions... pmatos C++ 6 04-26-2007 05:39 PM
Free memory allocate by a STL vector, vector of vector, map of vector Allerdyce.John@gmail.com C++ 8 02-18-2006 12:48 AM
Best way to write text content from servlet Bruce Lee Java 1 10-09-2005 08:41 AM
how the vector is created, how to pass vector to webservices method apachesoap:Vector Rushikesh Joshi Perl Misc 0 07-10-2004 01:04 PM



Advertisments