Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > How to read large amounts of output via popen

Reply
Thread Tools

How to read large amounts of output via popen

 
 
loial
Guest
Posts: n/a
 
      08-06-2010
I need to read a large amount of data that is being returned in
standard output by a shell script I am calling.

(I think the script should really be writing to a file but I have no
control over that)

Currently I have the following code. It seeems to work, however I
suspect this may not work with large amounts of standard output.

What is the best way to read a large amount of data from standard
output and write to a file?

Here is my code.

process=subprocess.Popen(['myscript', 'param1'],
shell=False,stdout=subprocess.PIPE,stderr=subproce ss.PIPE)

cmdoutput=process.communicate()

myfile = open('/home/john/myoutputfile','w')

myfile.write(cmdoutput[0])

myfile.close()

 
Reply With Quote
 
 
 
 
Gabriel Genellina
Guest
Posts: n/a
 
      08-06-2010
En Fri, 06 Aug 2010 06:06:29 -0300, loial <(E-Mail Removed)> escribió:

> I need to read a large amount of data that is being returned in
> standard output by a shell script I am calling.
>
> (I think the script should really be writing to a file but I have no
> control over that)
>
> Currently I have the following code. It seeems to work, however I
> suspect this may not work with large amounts of standard output.
>
> What is the best way to read a large amount of data from standard
> output and write to a file?
>
> Here is my code.
>
> process=subprocess.Popen(['myscript', 'param1'],
> shell=False,stdout=subprocess.PIPE,stderr=subproce ss.PIPE)
>
> cmdoutput=process.communicate()
>
> myfile = open('/home/john/myoutputfile','w')
>
> myfile.write(cmdoutput[0])
>
> myfile.close()



If all you do with the process' output is to write it to the output file,
you can avoid the intermediate step:


myfile = open('/home/john/myoutputfile','w')
myerror = open('/home/john/myerrorfile','w')
process=subprocess.Popen(['myscript', 'param1'],
shell=False,stdout=myfile,stderr=myerror)
process.wait()

(untested)

--
Gabriel Genellina

 
Reply With Quote
 
 
 
 
Nobody
Guest
Posts: n/a
 
      08-06-2010
On Fri, 06 Aug 2010 02:06:29 -0700, loial wrote:

> I need to read a large amount of data that is being returned in
> standard output by a shell script I am calling.
>
> (I think the script should really be writing to a file but I have no
> control over that)


If the script is writing to stdout, you get to decide whether its stdout
is a pipe, file, tty, etc.

> Currently I have the following code. It seeems to work, however I
> suspect this may not work with large amounts of standard output.


> process=subprocess.Popen(['myscript', 'param1'],
> shell=False,stdout=subprocess.PIPE,stderr=subproce ss.PIPE)
>
> cmdoutput=process.communicate()


It's certainly not the best way to read large amounts of output.
Unfortunately, better solutions get complicated when you need to read more
than one of stdout and stderr, or if you also need to write to stdin.

If you only need stdout, you can just read from process.stdout in a loop.
You can leave stderr going to wherever the script's stderr goes (e.g. the
terminal), or redirect it to a file.

If you really do need both stdout and stderr, then you either need to
enable non-blocking I/O, or use a separate thread for each stream, or
redirect at least one of them to a file.

FWIW, Popen.communicate() uses non-blocking I/O on Unix and separate
threads on Windows (the standard library doesn't include a mechanism to
enable non-blocking I/O on Windows).

> What is the best way to read a large amount of data from standard
> output and write to a file?


For this case, the best way is to just redirect stdout to a file, rather
than passing it through the script, i.e.:

outfile = open('outputfile', 'w')
process = subprocess.call(..., stdout = outfile)
outfile.close()

 
Reply With Quote
 
loial
Guest
Posts: n/a
 
      08-06-2010
Ok, thats great. Thanks for the very elegant solution(s)




On 6 Aug, 13:44, Nobody <(E-Mail Removed)> wrote:
> On Fri, 06 Aug 2010 02:06:29 -0700, loial wrote:
> > I need to read a large amount of data that is being returned in
> > standard output by a shell script I am calling.

>
> > (I think the script should really be writing to a file but I have no
> > control over that)

>
> If the script is writing to stdout, you get to decide whether its stdout
> is a pipe, file, tty, etc.
>
> > Currently I have the following code. It seeems to work, however I
> > suspect this may not work with large amounts of standard output.
> > process=subprocess.Popen(['myscript', 'param1'],
> > shell=False,stdout=subprocess.PIPE,stderr=subproce ss.PIPE)

>
> > cmdoutput=process.communicate()

>
> It's certainly not the best way to read large amounts of output.
> Unfortunately, better solutions get complicated when you need to read more
> than one of stdout and stderr, or if you also need to write to stdin.
>
> If you only need stdout, you can just read from process.stdout in a loop.
> You can leave stderr going to wherever the script's stderr goes (e.g. the
> terminal), or redirect it to a file.
>
> If you really do need both stdout and stderr, then you either need to
> enable non-blocking I/O, or use a separate thread for each stream, or
> redirect at least one of them to a file.
>
> FWIW, Popen.communicate() uses non-blocking I/O on Unix and separate
> threads on Windows (the standard library doesn't include a mechanism to
> enable non-blocking I/O on Windows).
>
> > What is the best way to read a large amount of data from standard
> > output and write to a file?

>
> For this case, the best way is to just redirect stdout to a file, rather
> than passing it through the script, i.e.:
>
> * * * * outfile = open('outputfile', 'w')
> * * * * process = subprocess.call(..., stdout = outfile)
> * * * * outfile.close()


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
HTTP POST to send large amounts of data? Bint HTML 1 03-19-2006 02:11 PM
transferring large amounts of data Andersen Java 2 11-03-2005 03:16 AM
What is the best email program for handling large amounts of incoming mail that needs to be filtered? C. B. Computer Support 19 04-13-2004 11:03 PM
Using Dataset for large amounts of data Brent ASP .Net 3 04-07-2004 01:43 PM



Advertisments