Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > locking files on Linux

Reply
Thread Tools

locking files on Linux

 
 
andrea crotti
Guest
Posts: n/a
 
      10-18-2012
I'm trying to understand how I can lock a file while writing on it,
because I might have multiple processes working on it at the same time.

I found the fcntl.lockf function but if I do this:

In [109]: locked = open('locked.txt', 'w')

In [110]: fcntl.lockf(locked, fcntl.LOCK_EX)

I can happily open the file with vim from somewhere and write on it, so
it doesn't seem to be very useful, or am I missing something?

I can otherwise use the classic ".lock" file when working on it, but I
would be glad to see a smarter solution (should only work on Linux)..
 
Reply With Quote
 
 
 
 
Grant Edwards
Guest
Posts: n/a
 
      10-18-2012
On 2012-10-18, andrea crotti <(E-Mail Removed)> wrote:

> I'm trying to understand how I can lock a file while writing on it,
> because I might have multiple processes working on it at the same time.
>
> I found the fcntl.lockf function but if I do this:
>
> In [109]: locked = open('locked.txt', 'w')
>
> In [110]: fcntl.lockf(locked, fcntl.LOCK_EX)
>
> I can happily open the file with vim from somewhere and write on it, so
> it doesn't seem to be very useful, or am I missing something?


File locks under Unix have historically been "advisory". That means
that programs have to _choose_ to pay attention to them. Most
programs do not.

Linux does support mandatory locking, but it's rarely used and must be
manually enabled at the filesystem level. It's probably worth noting
that in the Linux kernel docs, the document on mandatory file locking
begins with a section titled "Why you should avoid mandatory locking".

http://en.wikipedia.org/wiki/File_lo...x-like_systems
http://kernel.org/doc/Documentation/...tems/locks.txt
http://kernel.org/doc/Documentation/...ry-locking.txt
http://www.thegeekstuff.com/2012/04/...locking-types/
http://www.hackinglinuxexposed.com/a.../20030623.html

--
Grant Edwards grant.b.edwards Yow! Your CHEEKS sit like
at twin NECTARINES above
gmail.com a MOUTH that knows no
BOUNDS --
 
Reply With Quote
 
 
 
 
andrea crotti
Guest
Posts: n/a
 
      10-18-2012
2012/10/18 Grant Edwards <(E-Mail Removed)>:
> On 2012-10-18, andrea crotti <(E-Mail Removed)> wrote:
>
>
> File locks under Unix have historically been "advisory". That means
> that programs have to _choose_ to pay attention to them. Most
> programs do not.
>
> Linux does support mandatory locking, but it's rarely used and must be
> manually enabled at the filesystem level. It's probably worth noting
> that in the Linux kernel docs, the document on mandatory file locking
> begins with a section titled "Why you should avoid mandatory locking".
>
> http://en.wikipedia.org/wiki/File_lo...x-like_systems
> http://kernel.org/doc/Documentation/...tems/locks.txt
> http://kernel.org/doc/Documentation/...ry-locking.txt
> http://www.thegeekstuff.com/2012/04/...locking-types/
> http://www.hackinglinuxexposed.com/a.../20030623.html
>
> --
> Grant Edwards grant.b.edwards Yow! Your CHEEKS sit like
> at twin NECTARINES above
> gmail.com a MOUTH that knows no
> BOUNDS --
> --
> http://mail.python.org/mailman/listinfo/python-list



Uhh I see thanks, I guess I'll use the good-old .lock file (even if it
might have some problems too).

Anyway I'm only afraid that my same application could modify the
files, so maybe I can instruct it to check if the file is locked.

Or maybe using sqlite would work even if writing from different
processes?

I would prefer to keep something human readable as INI-format though,
rather then a sqlite file..

Thanks
 
Reply With Quote
 
Grant Edwards
Guest
Posts: n/a
 
      10-18-2012
On 2012-10-18, andrea crotti <(E-Mail Removed)> wrote:
> 2012/10/18 Grant Edwards <(E-Mail Removed)>:
>> On 2012-10-18, andrea crotti <(E-Mail Removed)> wrote:
>>
>> File locks under Unix have historically been "advisory". That means
>> that programs have to _choose_ to pay attention to them. Most
>> programs do not.
>>
>> Linux does support mandatory locking, but it's rarely used and must be
>> manually enabled at the filesystem level. It's probably worth noting
>> that in the Linux kernel docs, the document on mandatory file locking
>> begins with a section titled "Why you should avoid mandatory locking".

>
> Uhh I see thanks, I guess I'll use the good-old .lock file (even if
> it might have some problems too).
>
> Anyway I'm only afraid that my same application could modify the
> files, so maybe I can instruct it to check if the file is locked.


If what you're guarding against is multiple instances of your
application modifying the file, then either of the advisory file
locking schemes or the separate lock file should work fine.

--
Grant Edwards grant.b.edwards Yow! All this time I've
at been VIEWING a RUSSIAN
gmail.com MIDGET SODOMIZE a HOUSECAT!
 
Reply With Quote
 
Oscar Benjamin
Guest
Posts: n/a
 
      10-18-2012
On 18 October 2012 14:44, andrea crotti <(E-Mail Removed)> wrote:
> 2012/10/18 Grant Edwards <(E-Mail Removed)>:
>> On 2012-10-18, andrea crotti <(E-Mail Removed)> wrote:
>>
>>
>> File locks under Unix have historically been "advisory". That means
>> that programs have to _choose_ to pay attention to them. Most
>> programs do not.
>>
>> Linux does support mandatory locking, but it's rarely used and must be
>> manually enabled at the filesystem level. It's probably worth noting
>> that in the Linux kernel docs, the document on mandatory file locking
>> begins with a section titled "Why you should avoid mandatory locking".
>>
>> http://en.wikipedia.org/wiki/File_lo...x-like_systems
>> http://kernel.org/doc/Documentation/...tems/locks.txt
>> http://kernel.org/doc/Documentation/...ry-locking.txt
>> http://www.thegeekstuff.com/2012/04/...locking-types/
>> http://www.hackinglinuxexposed.com/a.../20030623.html
>>
>> --
>> Grant Edwards grant.b.edwards Yow! Your CHEEKS sit like
>> at twin NECTARINES above
>> gmail.com a MOUTH that knows no
>> BOUNDS --
>> --
>> http://mail.python.org/mailman/listinfo/python-list

>
>
> Uhh I see thanks, I guess I'll use the good-old .lock file (even if it
> might have some problems too).


I think you've misunderstood what Grant meant.

>
> Anyway I'm only afraid that my same application could modify the
> files, so maybe I can instruct it to check if the file is locked.


In that case fcntl will work for you. The point is that fcntl only
locks the file if all of the applications accessing the file use
fcntl. Any other application such as vim can simply ignore the fcntl
lock. Have a read of the links that Grant posted.

Did you try writing twice from the same application that uses fcntl?

> Or maybe using sqlite would work even if writing from different
> processes?


That would also work.


Oscar
 
Reply With Quote
 
andrea crotti
Guest
Posts: n/a
 
      10-18-2012
2012/10/18 Grant Edwards <(E-Mail Removed)>:
>
> If what you're guarding against is multiple instances of your
> application modifying the file, then either of the advisory file
> locking schemes or the separate lock file should work fine.
>
> --
> Grant Edwards grant.b.edwards Yow! All this time I've
> at been VIEWING a RUSSIAN
> gmail.com MIDGET SODOMIZE a HOUSECAT!
> --
> http://mail.python.org/mailman/listinfo/python-list


Ok so I tried a small example to see if I can make it fail, but this
below just works perfectly fine.

Maybe it's too fast and it release the file in time, but I would
expect it to take some time and fail instead..

import fcntl

from multiprocessing import Process

FILENAME = 'file.txt'


def long_text():
return ('some text' * (100 * 100))


class Locked:
def __init__(self, fileobj):
self.fileobj = fileobj

def __enter__(self):
# any problems here?
fcntl.lockf(self.fileobj, fcntl.LOCK_EX)
return self.fileobj

def __exit__(self, type, value, traceback):
fcntl.lockf(self.fileobj, fcntl.LOCK_UN)


def write_to_file():
with open(FILENAME, 'w') as to_lock:
with Locked(to_lock):
to_lock.write(long_text())


if __name__ == '__main__':
Process(target=write_to_file).start()
Process(target=write_to_file).start()
 
Reply With Quote
 
Oscar Benjamin
Guest
Posts: n/a
 
      10-18-2012
On 18 October 2012 15:49, andrea crotti <(E-Mail Removed)> wrote:
> 2012/10/18 Grant Edwards <(E-Mail Removed)>:
>>
>> If what you're guarding against is multiple instances of your
>> application modifying the file, then either of the advisory file
>> locking schemes or the separate lock file should work fine.

>
> Ok so I tried a small example to see if I can make it fail, but this
> below just works perfectly fine.
>
> Maybe it's too fast and it release the file in time, but I would
> expect it to take some time and fail instead..


Why not come up with a test that actually shows you if it works? Here
are two suggestions:

1) Use time.sleep() so that you know how long the lock is held for.
2) Write different data into the file from each process and see what
you end up with.

>
> import fcntl
>
> from multiprocessing import Process
>
> FILENAME = 'file.txt'
>
>
> def long_text():
> return ('some text' * (100 * 100))
>
>
> class Locked:
> def __init__(self, fileobj):
> self.fileobj = fileobj
>
> def __enter__(self):
> # any problems here?
> fcntl.lockf(self.fileobj, fcntl.LOCK_EX)
> return self.fileobj
>
> def __exit__(self, type, value, traceback):
> fcntl.lockf(self.fileobj, fcntl.LOCK_UN)
>
>
> def write_to_file():
> with open(FILENAME, 'w') as to_lock:


I don't think it will work if you truncate the file like this. This
will empty the file *before* checking for the lock. Try opening the
file for reading and writing (without truncating).

> with Locked(to_lock):
> to_lock.write(long_text())
>
>
> if __name__ == '__main__':
> Process(target=write_to_file).start()
> Process(target=write_to_file).start()



Oscar
 
Reply With Quote
 
andrea crotti
Guest
Posts: n/a
 
      10-18-2012
2012/10/18 Oscar Benjamin <(E-Mail Removed)>:
>
> Why not come up with a test that actually shows you if it works? Here
> are two suggestions:
>
> 1) Use time.sleep() so that you know how long the lock is held for.
> 2) Write different data into the file from each process and see what
> you end up with.
>



Ok thanks I will try, but I thought that what I did was the worst
possible case, because I'm opening and writing on the same file from
two different processes, locking the file with LOCK_EX.

It should not open it at all as far as I understood...
 
Reply With Quote
 
Oscar Benjamin
Guest
Posts: n/a
 
      10-18-2012
On 18 October 2012 16:08, andrea crotti <(E-Mail Removed)> wrote:
> 2012/10/18 Oscar Benjamin <(E-Mail Removed)>:
>>
>> Why not come up with a test that actually shows you if it works? Here
>> are two suggestions:
>>
>> 1) Use time.sleep() so that you know how long the lock is held for.
>> 2) Write different data into the file from each process and see what
>> you end up with.
>>

>
>
> Ok thanks I will try, but I thought that what I did was the worst
> possible case, because I'm opening and writing on the same file from
> two different processes, locking the file with LOCK_EX.
>
> It should not open it at all as far as I understood...


I don't think you have understood. Read the link that Grant posted:
http://en.wikipedia.org/wiki/File_lo...x-like_systems

And my other comment:

>> def write_to_file():
>> with open(FILENAME, 'w') as to_lock:

>
> I don't think it will work if you truncate the file like this. This
> will empty the file *before* checking for the lock. Try opening the
> file for reading and writing (without truncating).


The lock is cooperative. It does not prevent the file from being
opened or overwritten. It only prevents any other process from
obtaining the lock. Here you open the file with mode 'w' which
truncates the file instantly (without checking for the lock).


Oscar
 
Reply With Quote
 
andrea crotti
Guest
Posts: n/a
 
      10-19-2012
2012/10/18 Oscar Benjamin <(E-Mail Removed)>:
>
> The lock is cooperative. It does not prevent the file from being
> opened or overwritten. It only prevents any other process from
> obtaining the lock. Here you open the file with mode 'w' which
> truncates the file instantly (without checking for the lock).
>
>
> Oscar



Very good thanks now I understood, actually my problem was in the
assumption that it should fail when the lock is already taken, but by
default lockf just blocks until the lock is released.

It seems to work quite nicely so I'm going to use this..
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: ATTN: Aardvark they are locking in on your Linux security blanket.You can run, but you can't hide from it. Mike Easter Computer Support 5 04-22-2010 04:21 PM
Re: ATTN: Aardvark they are locking in on your Linux securityblanket. You can run, but you can't hide from it. Aardvark Computer Support 3 04-22-2010 02:20 PM
Locking locking resolution Frontpage raiderhawk General Computer Support 0 01-08-2008 01:42 AM
increase in cpu usage on locking and locking the system sowmya.rangineni@gmail.com Computer Support 0 06-15-2007 12:06 PM
Application locking to support optimisitc locking ? Timasmith Java 4 11-01-2006 12:42 AM



Advertisments