Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > copyfile avoiding overwrites and race conditions

Reply
Thread Tools

copyfile avoiding overwrites and race conditions

 
 
manuelg@gmail.com
Guest
Posts: n/a
 
      06-16-2006
Here is a code fragment, where I am trying to copy a file, avoiding
overwrites and race conditions. The filename gets a '02','03','04' etc
appended to the end if a file with that name already exists.

I know the writing of the single space is overkill, but I am surprised
I cannot find an example of this floating around the web and newsgroups

my understanding of 'os.open' and 'os.fdopen' is minimal

## start fragment

i = 0
while True:

if i >= 100: raise RanOutFileNamesError(fullpath)
i += 1

if i > 1:
name0 = '%s_%02i%s' % (suggested_name, i, file_extension)
else:
assert i == 1
name0 = '%s%s' % (suggested_name, file_extension)
fullpath = os.path.join(path0, name0)

# create dummy file, force overwrite

try:
fd = os.open(
fullpath,
os.O_CREAT | os.O_EXCL | os.O_WRONLY)
except OSError:
continue

# is this really necessary?
file = os.fdopen(fd, "w")
file.write(' ')
file.close()

shutil.copyfile(original_filepath, fullpath)

return fullpath

## end fragment

 
Reply With Quote
 
 
 
 
Larry Bates
Guest
Posts: n/a
 
      06-19-2006
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> Here is a code fragment, where I am trying to copy a file, avoiding
> overwrites and race conditions. The filename gets a '02','03','04' etc
> appended to the end if a file with that name already exists.
>
> I know the writing of the single space is overkill, but I am surprised
> I cannot find an example of this floating around the web and newsgroups
>
> my understanding of 'os.open' and 'os.fdopen' is minimal
>
> ## start fragment
>
> i = 0
> while True:
>
> if i >= 100: raise RanOutFileNamesError(fullpath)
> i += 1
>
> if i > 1:
> name0 = '%s_%02i%s' % (suggested_name, i, file_extension)
> else:
> assert i == 1
> name0 = '%s%s' % (suggested_name, file_extension)
> fullpath = os.path.join(path0, name0)
>
> # create dummy file, force overwrite
>
> try:
> fd = os.open(
> fullpath,
> os.O_CREAT | os.O_EXCL | os.O_WRONLY)
> except OSError:
> continue
>
> # is this really necessary?
> file = os.fdopen(fd, "w")
> file.write(' ')
> file.close()
>
> shutil.copyfile(original_filepath, fullpath)
>
> return fullpath
>
> ## end fragment
>

I guess my approach would be different. To eliminate any race
conditions, I would keep a small text file that always contained
the next filename that is to be written. Something like:

nextfiletowrite=/path/filename006.dat

I would try to get a lock on this file, read it, extract next
filename, increment the counter portion of the filename,
write it back out and unlock it. Now I have the name of the
file to write that is unique to my instance and I can write it
without worrying about other processes.

Hope this helps.

-Larry Bates
 
Reply With Quote
 
 
 
 
manuelg@gmail.com
Guest
Posts: n/a
 
      06-19-2006
Larry Bates wrote:

> I guess my approach would be different. To eliminate any race
> conditions, I would keep a small text file that always contained
> the next filename that is to be written. Something like:
>
> nextfiletowrite=/path/filename006.dat
>
> I would try to get a lock on this file, read it, extract next
> filename, increment the counter portion of the filename,
> write it back out and unlock it. Now I have the name of the
> file to write that is unique to my instance and I can write it
> without worrying about other processes.


Yes, that would work as well

but I get the feeling that

1) most people don't write to an automatically incrementing filename
2) the few who do don't sweat the race condition

Thanks for your reply

Manuel

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Drop folder and race conditions Larry Bates Python 1 10-09-2007 01:39 PM
How to prevent from race conditions to share data between many process and thread in python mars Python 6 02-07-2007 08:52 AM
Newbie Q: XMLHttpRequest.send() and race conditions? kj Javascript 4 12-22-2006 12:37 PM
race conditions/pulse width The Weiss Family VHDL 6 10-19-2004 02:47 PM
Mega Pixel race is like the Mhz Race Hugo Drax Digital Photography 7 01-12-2004 11:07 AM



Advertisments