«

Hi David and Matt. I appreciate your help which has got me moving
forward again so many thanks for your reply. I have been using
subprocess.Popen a fair bit but this was the first time I had to use
subprocess to capture large file output. The trouble I was having was
with the process would just hang. Chunking was the solution. I guess I
assumed this would be taken care of in the internals.

Overall, I wish subprocess had some better documentation since it is
definitely not a drop in replacement for os.system. In other
circumstances I am using subprocess.call() for simple calls which works
fine.

The speed of this solution is slower than os.system. Would a queue of
some kind be needed to speed this up? Has anyone implemented something
like this? Many thanks.

Regards,
David

Matt Nordhoff wrote:
> David Pratt wrote:
>> Hi. I am trying to replace a system call with a subprocess call. I have
>> tried subprocess.Popen and subprocess.call with but have not been
>> successful. The command line would be:
>>
>> svnadmin dump /my/repository > svndump.db
>>
>> This is what I am using currently:
>>
>> os.system('svnadmin dump %s > %s' % (svn_dir,
>> os.path.join(backup_dir, 'svndump.db')))
>>
>> Many thanks.
>
> Try this:
>
> import os.path
> import subprocess
>
> p = subprocess.Popen(
> ['svnadmin', 'dump', svndir],
> stdout=subprocess.PIPE,
> )
>
> fh = open(os.path.join(backup_dir, 'svndump.db'), 'wb')
>
> while True:
> chunk = p.stdout.read(2**20) # 1 MB
> if not chunk:
> break
> fh.write(chunk)
>
> fh.close()
>
> It reads svnadmin's stdout in 1 MB chunks, in case it's large enough
> that reading the whole thing into RAM at once would be a bad idea.
>
> No error handling. For one, you might want to add a try...finally to
> ensure that fh will get closed. (Or if you have Python 2.5, use a with
> statement! ) Also, Popen will raise an OSError if svnadmin can't be
> found or something. And this isn't even considering svnadmin erroring out...
>
> svnadmin's stderr will go to your stderr.
>
> I didn't test it, but I'm pretty sure it will work. (I spotted a syntax
> error while writing that though.) I don't have much experience with
> Popen's stdio objects, so it's possible you'd need to do something like
> call p.wait() to wait for it to exit before being able to read its stdout.
>
> It could be slower than the os.system version, since now Python is doing
> all of the I/O, instead of your shell, but I doubt that'll be a big problem.
>
> (Also, insert suggestion about using a good VCS. )