Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > filecmp.dircmp performance

Reply
Thread Tools

filecmp.dircmp performance

 
 
dads
Guest
Posts: n/a
 
      01-08-2011
I'm creating a one way sync program, it's to automate backing up data
over the wan from our shops to a server at head office. It uses
filecmp.dircmp() but the performance seems poor to me.

for x in dc.diff_files:
srcfp = os.path.join(src, x)
self.fn777(srcfp)
if os.path.isfile(srcfp):
try:
shutil.copy2(srcfp, dst)
self.lg.add_diffiles(src, x)
except Exception, e:
self.lg.add_errors(e)

I tested it at a store which is only around 50 miles away on a 10Mbps
line, the directory has 59 files that are under 100KB. When it gets to
dc.diff_files it takes 15mins to complete. Looking at the filecmp.py
it's only using os.stat, it seems excessively long.

code:
http://pastebin.com/QskXGDQT
 
Reply With Quote
 
 
 
 
Peter Otten
Guest
Posts: n/a
 
      01-08-2011
dads wrote:

> I'm creating a one way sync program, it's to automate backing up data
> over the wan from our shops to a server at head office. It uses
> filecmp.dircmp() but the performance seems poor to me.
>
> for x in dc.diff_files:
> srcfp = os.path.join(src, x)
> self.fn777(srcfp)
> if os.path.isfile(srcfp):
> try:
> shutil.copy2(srcfp, dst)
> self.lg.add_diffiles(src, x)
> except Exception, e:
> self.lg.add_errors(e)
>
> I tested it at a store which is only around 50 miles away on a 10Mbps
> line, the directory has 59 files that are under 100KB. When it gets to
> dc.diff_files it takes 15mins to complete. Looking at the filecmp.py
> it's only using os.stat, it seems excessively long.


As a baseline it would be interesting to see how long it takes to copy those
59 files using system tools.

However, there are efficient tools out there that work hard to reduce the
traffic over the net which is likely to be the bottleneck. I suggest that
you have have a look at

http://en.wikipedia.org/wiki/Rsync
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Performance Tutorials Services - Boosting Performance by DisablingUnnecessary Services on Windows XP Home Edition Software Engineer Javascript 0 06-10-2011 02:18 AM
OCZ Rally High Performance USB2 (Dual Channel) Flash Memory Silverstrand Front Page News 0 09-20-2005 03:35 AM
How to quickly improve your computer's performance @ Bona... Silverstrand Front Page News 0 08-24-2005 01:26 PM
HEXUS.review: Rockdirect XTI 3.8 Performance Silverstrand Front Page News 0 08-08-2005 02:37 PM
Web Form Performance Versus Single File Performance jm ASP .Net 1 12-12-2003 11:14 PM



Advertisments