Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Avoiding excessive writes

Reply
Thread Tools

Avoiding excessive writes

 
 
Gabriel Genellina
Guest
Posts: n/a
 
      03-19-2009
En Thu, 19 Mar 2009 00:52:54 -0200, <(E-Mail Removed)> escribió:

> I'm aware that C, and thus Python I suppose, will cache file writes
> for efficiency and to avoid excessive disk activity,


Python 2.X, yes, because it uses C file streams (FILE) to implement Python
file objects. Python 3.x avoids that layer.

> Does the cache get written simply when an X-byte buffer fills up, and
> can I change X?


Yes, use the bufsize argument to open.

> Does it write after waiting so long?


I don't think so. You may call flush() yourself after some time.

--
Gabriel Genellina

 
Reply With Quote
 
 
 
 
prometheus235@gmail.com
Guest
Posts: n/a
 
      03-19-2009
I'm aware that C, and thus Python I suppose, will cache file writes
for efficiency and to avoid excessive disk activity, causing
occasional problems when forgetting to flush. How exactly this works
is still somewhat of a mystery to me, but a concern because of a
program I'm developing:

The program performs tests on some hardware attached to a serial port
and cycles every few seconds; for the reliability tests I'll do this
anywhere from 10k to 100k times, logging each event. Sometimes the
file gets targeted to a thumbdrive, and 100k writes likely puts
significant wear on the flash, so what I've done in the past is just
make my own cache of ~500 data points then go and write it all at
once, but is there some way to set up the file operations so it
effectively does this for me?

Does the cache get written simply when an X-byte buffer fills up, and
can I change X? Does it write after waiting so long?
 
Reply With Quote
 
 
 
 
John Machin
Guest
Posts: n/a
 
      03-19-2009
On Mar 19, 1:20*pm, "Gabriel Genellina" <(E-Mail Removed)>
wrote:
> En Thu, 19 Mar 2009 00:52:54 -0200, <(E-Mail Removed)> escribió:
>
> > I'm aware that C, and thus Python I suppose, will cache file writes
> > for efficiency and to avoid excessive disk activity,

>
> Python 2.X, yes, because it uses C file streams (FILE) to implement Python *
> file objects. Python 3.x avoids that layer.
>
> > Does the cache get written simply when an X-byte buffer fills up, and
> > can I change X?

>
> Yes, use the bufsize argument to open.
>
> > Does it write after waiting so long?

>
> I don't think so. You may call flush() yourself after some time.
>


After it leaves Python, and the C stdio if that's used, isn't it then
at the whim of the operating system exactly when and how much gets
written to disk? My understanding is that this layer sometimes makes
an effort to detect that the client is writing size X blocks
sequentially and may decide to to buffer them up and do an N*X
physical write later (like where N * X might represent a whole number
of tracks) ... and/or you've got the drive itself which may do some
buffering ...

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
best practice to avoiding excessive memory usage?? Chris Perl Misc 9 11-20-2006 11:05 AM
Synchronous writes to a file Erick Sasse ASP .Net 10 10-07-2005 10:59 PM
download writes html code to file =?Utf-8?B?QUNhdW50ZXI=?= ASP .Net 3 01-11-2005 07:29 PM
how adoStream writes from asp to asp.net? Ben ASP .Net 8 08-19-2004 08:18 AM
Lots of Response.Writes of HTML - How do YOU do it? TinyTim ASP .Net 2 08-10-2004 03:22 AM



Advertisments