Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > fast copying of large files in python

Reply
Thread Tools

fast copying of large files in python

 
 
Catherine Moroney
Guest
Posts: n/a
 
      11-02-2011
Hello,

I'm working on an application that as part of its processing, needs to
copy 50 Meg binary files from one NFS mounted disk to another.

The simple-minded approach of shutil.copyfile is very slow, and I'm
guessing that this is due to the default 16K buffer size. Using
shtil.copyfileobj is faster when I set a larger buffer size, but it's
still slow compared to a "os.system("cp %s %s" % (f1,f2))" call.
Is this because of the overhead entailed in having to open the files
in copyfileobj?

I'm not worried about portability, so should I just do the
os.system call as described above and be done with it? Is that the
fastest method in this case? Are there any "pure python" ways of
getting the same speed as os.system?

Thanks,

Catherine
 
Reply With Quote
 
 
 
 
Dave Angel
Guest
Posts: n/a
 
      11-02-2011
On 11/02/2011 03:31 PM, Catherine Moroney wrote:
> Hello,
>
> I'm working on an application that as part of its processing, needs to
> copy 50 Meg binary files from one NFS mounted disk to another.
>
> The simple-minded approach of shutil.copyfile is very slow, and I'm
> guessing that this is due to the default 16K buffer size. Using
> shtil.copyfileobj is faster when I set a larger buffer size, but it's
> still slow compared to a "os.system("cp %s %s" % (f1,f2))" call.
> Is this because of the overhead entailed in having to open the files
> in copyfileobj?
>
> I'm not worried about portability, so should I just do the
> os.system call as described above and be done with it? Is that the
> fastest method in this case? Are there any "pure python" ways of
> getting the same speed as os.system?
>
> Thanks,
>
> Catherine


I suspect the fastest way would be to use scp, and not try to use your
local mount for either file. There are some options on scp that tell it
to just connect the (up to) two machines to each other and transfer,
where it won't even come to your local machine.

I do that sort of thing when I'm connecting over slow internet (and vpn)
to two machines that are on the same local subnet.

--

DaveA

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
server generates Delayed Write errors copying Very large Files Phil Lewis Windows 64bit 17 09-29-2010 05:03 PM
Reboot while copying large files between hard drives pegasus1919 Computer Support 4 05-10-2007 02:56 AM
Parsing large XML files FAST PedroX XML 9 06-27-2005 11:38 PM
Backing Up Large Files..Or A Large Amount Of Files Scott D. Weber For Unuathorized Thoughts Inc. Computer Support 1 09-19-2003 07:28 PM
Python is darn fast (was: How fast is Python) Michele Simionato Python 13 08-27-2003 03:58 AM



Advertisments