Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > paralell ftp uploads and pool size

Reply
Thread Tools

paralell ftp uploads and pool size

 
 
ben@bbarker.co.uk
Guest
Posts: n/a
 
      01-09-2013
Hello,

I have a python script that uploads multiple files from the local machine to a remote server in parallel via ftp using p process pool:

p = Pool(processes=x)

Now as I increase the value of x, the overall upload time for all files drops as expected. If I set x too high however, then an exception is thrown. The exact value at which this happens varies, but is ~20

Traceback (most recent call last):
File "uploadFTP.py", line 59, in <module>
FTP_Upload().multiupload()
File "uploadFTP.py", line 56, in multiupload
p.map(upload_function,files)
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 148, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 422, in get
raise self._value
EOFError

Now this is not a problem - 20 is more than enough - but I'm trying to understand the mechanisms involved, and why the exact number of processes at which this exception occurs seems to vary.

I guess it comes down to the current resources of the server itself...but any insight would be much appreciated!
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
PIX 501 issue routing between VPN pool and local pool eostrike Cisco 3 10-24-2008 09:43 PM
FTP uploads cut off at 25MB WCH Computer Support 12 03-25-2006 02:08 PM
Net::FTP problems getting files from Windows FTP server, but not Linux FTP Server. D. Buck Perl Misc 2 06-29-2004 02:05 PM
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. Guoqi Zheng ASP .Net 4 06-03-2004 06:39 PM
FTP uploads via browser takes path of desktop machine jim arnold HTML 1 03-05-2004 12:40 AM



Advertisments