Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Flushing buffer on file copy on linux

Reply
Thread Tools

Re: Flushing buffer on file copy on linux

 
 
Antoine Pitrou
Guest
Posts: n/a
 
      08-15-2012
J <dreadpiratejeff <at> gmail.com> writes:
>
> Now, the problem I have is that linux tends to buffer data writes to a
> device, and I want to work around that. When run in normal non-stress
> mode, the program is slow enough that the linux buffers flush and put
> the file on disk before the hash occurs. However, when run in stress
> mode, what I'm finding is that it appears that the files are possibly
> being hashed while still in the buffer, before being flushed to disk.


Your analysis is partly wrong. It is right that the files can be hashed from
in-memory buffers; but even if you flush the buffers to disk using standard
techniques (such as fsync()), those buffers still exist in memory, and
therefore the file will still be hashed from memory (for obvious efficiency
reasons).

I don't think there's a portable solution to get away entirely with the
in-memory buffers, but under Linux you can write "1" to the special file
/proc/sys/vm/drop_caches:

$ sudo sh -c "echo 1 > /proc/sys/vm/drop_caches"

Or, to quote the /proc man page:

/proc/sys/vm/drop_caches (since Linux 2.6.16)
Writing to this file causes the kernel to drop clean
caches, dentries and inodes from memory, causing that mem‐
ory to become free.

To free pagecache, use echo 1 > /proc/sys/vm/drop_caches;
to free dentries and inodes, use echo 2 >
/proc/sys/vm/drop_caches; to free pagecache, dentries and
inodes, use echo 3 > /proc/sys/vm/drop_caches.

Because this is a nondestructive operation and dirty
objects are not freeable, the user should run sync(
first.


Regards

Antoine.


--
Software development and contracting: http://pro.pitrou.net


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Flushing buffer on file copy on linux Cameron Simpson Python 0 08-15-2012 03:09 AM
Flushing buffer on file copy on linux J Python 0 08-15-2012 02:55 AM
Decorators and buffer flushing Ethan Metsger Python 0 02-28-2008 05:49 PM
Accelerated C++ - Chapter 1, confusion in understanding "flushing the buffer" arnuld C++ 5 03-18-2007 11:14 AM
Flushing the Buffer in .Net Eric ASP .Net 3 04-25-2004 11:23 PM



Advertisments