Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > timing issue: shutil.rmtree and os.makedirs

Reply
Thread Tools

timing issue: shutil.rmtree and os.makedirs

 
 
Tim
Guest
Posts: n/a
 
      07-29-2013
I have the following function (Python2.7 on FreeBSD) that results in an OSError.

My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path:

def make_clean_dir(directory):
if os.path.exists(directory):
if os.path.isdir(directory):
shutil.rmtree(directory)
else:
os.unlink(directory)
os.makedirs(directory)

The last bit of the traceback is:
File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir
os.makedirs(directory)
File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '/users/tim/testing/testing_html'

The directory 'testing_html' existed when I executed the function;
So I suppose the directory wasn't finished being removed by the time os.makedirs was invoked. How can avoid this? (A context manager maybe?).

thanks,
--Tim

 
Reply With Quote
 
 
 
 
Chris Angelico
Guest
Posts: n/a
 
      07-29-2013
On Mon, Jul 29, 2013 at 8:16 PM, Tim <(E-Mail Removed)> wrote:
> My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path:
>
> def make_clean_dir(directory):
> if os.path.exists(directory):
> if os.path.isdir(directory):
> shutil.rmtree(directory)
> else:
> os.unlink(directory)
> os.makedirs(directory)
>
> The last bit of the traceback is:
> File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir
> os.makedirs(directory)
> File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
> mkdir(name, mode)
> OSError: [Errno 17] File exists: '/users/tim/testing/testing_html'
>
> The directory 'testing_html' existed when I executed the function;


First thing I'd check is: Did rmtree succeed? Try removing the
makedirs and test it again; then, when your process has completely
finished, see if the directory is there. If it is, the problem is in
rmtree - for instance:

* You might not have permission to remove everything
* There might be a messed-up object in the file system
* If the directory is a remote share mount point, the other end might
have lied about the removal
* Something might have been created inside the directory during the removal
* Myriad other possibilities

As I understand rmtree's docs, any errors *that it detects* will be
raised as exceptions (since you haven't told it to suppress or handle
them), but possibly there's an error that it isn't able to detect.
Worth a test, anyhow.

ChrisA
 
Reply With Quote
 
 
 
 
Tim
Guest
Posts: n/a
 
      07-30-2013
On Monday, July 29, 2013 7:52:36 PM UTC-4, Chris Angelico wrote:
> On Mon, Jul 29, 2013 at 8:16 PM, Tim wrote:
> > My intent is to pass it a directory name or path and if it exists, use shutil.rmtree to remove whatever is there (if it isn't a directory, try to unlink it); then use os.makedirs to create a new directory or path:


> > def make_clean_dir(directory):
> > if os.path.exists(directory):
> > if os.path.isdir(directory):
> > shutil.rmtree(directory)
> > else:
> > os.unlink(directory)
> > os.makedirs(directory)
> >
> > The last bit of the traceback is:
> > File "/develop/myproject/helpers/__init__.py", line 35, in make_clean_dir
> > os.makedirs(directory)
> > File "/usr/local/lib/python2.7/os.py", line 157, in makedirs
> > mkdir(name, mode)
> > OSError: [Errno 17] File exists: '/users/tim/testing/testing_html'
> >
> > The directory 'testing_html' existed when I executed the function;

>
> First thing I'd check is: Did rmtree succeed? Try removing the
> makedirs and test it again; then, when your process has completely
> finished, see if the directory is there. If it is, the problem is in
> rmtree - for instance:


> * You might not have permission to remove everything
> * There might be a messed-up object in the file system
> * If the directory is a remote share mount point, the other end might
> have lied about the removal
> * Something might have been created inside the directory during the removal
> * Myriad other possibilities
> As I understand rmtree's docs, any errors *that it detects* will be
> raised as exceptions (since you haven't told it to suppress or handle
> them), but possibly there's an error that it isn't able to detect.
> Worth a test, anyhow.
>
> ChrisA


Thanks Chris, but the directory was actually removed on the first run in spite of the traceback; when I run it a second time (immediately after the first time), it runs fine. That's why I thought it was a timing issue. I thought about just putting a sleep in there, but that made me feel dirty.

hmm, now that you mention it, this is executing on a remote box with accessto the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote.

But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished.

thanks,
--Tim
 
Reply With Quote
 
Chris Angelico
Guest
Posts: n/a
 
      07-30-2013
On Tue, Jul 30, 2013 at 2:10 PM, Tim <(E-Mail Removed)> wrote:
> hmm, now that you mention it, this is executing on a remote box with access to the same file system my local calling program is on. That is, there is a local call to an intermediate script that connects to a socket on the remote where the above program actually runs, but the file system is the same place for both local and remote.
>
> But even so, since the script that does the rmtree and mkdir is running on the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished.


Hmm. What system is used for the file system sharing? I know quite a
few of them lie about whether something's been completely done or not.

Can you use inotify to tell you when the directory's been deleted?
Seems stupid though.

Worst case, all you need is a quick loop at the bottom, eg:

for delay in 100,300,600,1000,3000,5000,10000:
if not os.path.exists(directory): break
sleep(delay)

That'll sleep a maximum of 20 seconds, tune as required. Of course, if
there's a way to tune the FS to guarantee that the removal blocks
correctly, that would be way better than sleep()!

ChrisA
 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      07-30-2013
On Tue, 30 Jul 2013 14:27:10 +0100, Chris Angelico wrote:

> for delay in 100,300,600,1000,3000,5000,10000:
> if not os.path.exists(directory): break
> sleep(delay)
>
> That'll sleep a maximum of 20 seconds, tune as required.


Actually, that will sleep a maximum of 5.55 hours, and a minimum of 1.7
minutes (assuming the directory doesn't get deleted instantaneously).

time.sleep() takes an argument in seconds.



--
Steven
 
Reply With Quote
 
Chris Angelico
Guest
Posts: n/a
 
      07-30-2013
On Tue, Jul 30, 2013 at 3:07 PM, Steven D'Aprano
<(E-Mail Removed)> wrote:
> On Tue, 30 Jul 2013 14:27:10 +0100, Chris Angelico wrote:
>
>> for delay in 100,300,600,1000,3000,5000,10000:
>> if not os.path.exists(directory): break
>> sleep(delay)
>>
>> That'll sleep a maximum of 20 seconds, tune as required.

>
> Actually, that will sleep a maximum of 5.55 hours, and a minimum of 1.7
> minutes (assuming the directory doesn't get deleted instantaneously).
>
> time.sleep() takes an argument in seconds.


LOL! Whoops. That's what I get for not checking my docs. This is why
we have public responses, my errors can be caught by someone else.

ChrisA
 
Reply With Quote
 
Tim
Guest
Posts: n/a
 
      07-30-2013
On Tuesday, July 30, 2013 9:27:10 AM UTC-4, Chris Angelico wrote:
> On Tue, Jul 30, 2013 at 2:10 PM, Tim wrote:
> > hmm, now that you mention it, this is executing on a remote box with access to the same file system my local calling program is on. That is, thereis a local call to an intermediate script that connects to a socket on theremote where the above program actually runs, but the file system is the same place for both local and remote.
> >
> > But even so, since the script that does the rmtree and mkdir is runningon the same machine (even though it's remote), I would think the mkdir couldn't execute until the rmtree was completely finished.

>
> Hmm. What system is used for the file system sharing? I know quite a
> few of them lie about whether something's been completely done or not.
> Can you use inotify to tell you when the directory's been deleted?
> Seems stupid though.
> Worst case, all you need is a quick loop at the bottom, eg:
>

<<snip, thanks for the code and correction>>
> ChrisA


Argg, this isn't the first time I've had troubles with the file system. This is FreeBSD and NFS. I will code up a progressive delay as you mentioned (with Steve's correction).

thanks much!
--Tim


 
Reply With Quote
 
Chris Angelico
Guest
Posts: n/a
 
      07-30-2013
On Tue, Jul 30, 2013 at 4:37 PM, Tim <(E-Mail Removed)> wrote:
> Argg, this isn't the first time I've had troubles with the file system. This is FreeBSD and NFS. I will code up a progressive delay as you mentioned (with Steve's correction).


I've used several different networked file systems, including
NetBIOS/NetBEUI/SMB/Samba/etc, sshfs/cifs, and nfs. Not one of them
"feels" as clean as I'd like, though sshfs comes closest. There always
seem to be hacks around.

ChrisA
 
Reply With Quote
 
Göktuğ Kayaalp
Guest
Posts: n/a
 
      07-30-2013
On Jul 30, 2013 3:29 PM, "Chris Angelico" <(E-Mail Removed)> wrote:
>
> On Tue, Jul 30, 2013 at 2:10 PM, Tim <(E-Mail Removed)> wrote:
> > hmm, now that you mention it, this is executing on a remote box with

access to the same file system my local calling program is on. That is,
there is a local call to an intermediate script that connects to a socket
on the remote where the above program actually runs, but the file system is
the same place for both local and remote.
> >
> > But even so, since the script that does the rmtree and mkdir is running

on the same machine (even though it's remote), I would think the mkdir
couldn't execute until the rmtree was completely finished.
>
> Hmm. What system is used for the file system sharing? I know quite a
> few of them lie about whether something's been completely done or not.
>
> Can you use inotify to tell you when the directory's been deleted?
> Seems stupid though.


Inotify is a linux thing, but there is kqueue for Free?BSD. OP can run the
deletion procedure, wait for a NOTE_DELETE event, which would block, and
create the fresh directory afterwards. It may require some C hacking
though, in case Python lacks a kqueue wrapper. I think that this kind of
approach would be more sound than a check-wait-loop approach.

(I would elaborate more with pointers to appropriate documentation, but I'm
on a silly tablet, please excuse me for that.)

-gk

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Why is the shutil module called shutil? Fletcher Johnson Python 11 09-26-2011 04:18 AM
Re: shutil.rmtree raises "OSError: [Errno 39] Directory not empty"exception Tim Chase Python 1 07-13-2009 12:03 AM
A faster shutil.rmtree or maybe a command. martijn@gamecreators.nl Python 2 10-11-2005 10:01 AM
using wildcards in rmtree or blog on windows Chuck Bradley Perl Misc 2 07-16-2004 04:23 AM
rmtree and untaint Flagstaff Perl Misc 4 01-05-2004 02:12 AM



Advertisments