Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > How to send a var to stdin of an external software

Reply
Thread Tools

How to send a var to stdin of an external software

 
 
Benjamin Watine
Guest
Posts: n/a
 
      03-13-2008
Hi the list,

I need to send a var to stdin of an external soft ("cat" command for
example).

How can I do this ? I would like a function like that :

theFunction ('cat -', stdin=myVar)

I don't need to get any return value.

Another related question : Is there's a limitation of var size ? I would
have var up to 10 MB.

Thanks !

Ben

 
Reply With Quote
 
 
 
 
Marko Rauhamaa
Guest
Posts: n/a
 
      03-13-2008
Benjamin Watine <(E-Mail Removed)>:

> How can I do this ? I would like a function like that :
>
> theFunction ('cat -', stdin=myVar)
>
> Another related question : Is there's a limitation of var size ? I
> would have var up to 10 MB.


import subprocess
myVar = '*' * 10000000
cat = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE)
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()


Marko

--
Marko Rauhamaa (E-Mail Removed) http://pacujo.net/marko/
 
Reply With Quote
 
 
 
 
Sion Arrowsmith
Guest
Posts: n/a
 
      03-13-2008
Benjamin Watine <(E-Mail Removed)> wrote:
>How can I do this ? I would like a function like that :
>
> theFunction ('cat -', stdin=myVar)
>
>I don't need to get any return value.


http://docs.python.org/lib/node534.html says this is spelt

myVar = subprocess.Popen(["cat", "-"], stdout=subprocess.PIPE).communicate()[0]

(Probably not obvious how to find this if you've not come across the
backtick notation in shell or Perl.)

--
\S -- http://www.velocityreviews.com/forums/(E-Mail Removed) -- http://www.chaos.org.uk/~sion/
"Frankly I have no feelings towards penguins one way or the other"
-- Arthur C. Clarke
her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump
 
Reply With Quote
 
Benjamin Watine
Guest
Posts: n/a
 
      03-13-2008
Marko Rauhamaa a écrit :
> Benjamin Watine <(E-Mail Removed)>:
>
>> How can I do this ? I would like a function like that :
>>
>> theFunction ('cat -', stdin=myVar)
>>
>> Another related question : Is there's a limitation of var size ? I
>> would have var up to 10 MB.

>
> import subprocess
> myVar = '*' * 10000000
> cat = subprocess.Popen('cat',shell = True,stdin = subprocess.PIPE)
> cat.stdin.write(myVar)
> cat.stdin.close()
> cat.wait()
>
>
> Marko
>


Thank you Marko, it's exactly what I need.

And if somebody need it : to get the stdout in a var (myNewVar), not in
the shell :

cat = subprocess.Popen('cat', shell = True, stdin = subprocess.PIPE,
stdout=subprocess.PIPE)
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()
myNewVar = cat.stdout.read()

Is it correct ?

Ben
 
Reply With Quote
 
Bryan Olson
Guest
Posts: n/a
 
      03-13-2008
Benjamin Watine wrote:
> And if somebody need it : to get the stdout in a var (myNewVar), not in
> the shell :
>
> cat = subprocess.Popen('cat', shell = True, stdin = subprocess.PIPE,
> stdout=subprocess.PIPE)
> cat.stdin.write(myVar)
> cat.stdin.close()
> cat.wait()
> myNewVar = cat.stdout.read()
>
> Is it correct ?


No, not really. It is prone to deadlock. The external program might
work by iteratively reading a little input and writing a little
output, as 'cat' almost surely does. If the size of myVar exceeds
the buffer space in cat and the pipes, you get stuck.

Your Python program can block at "cat.stdin.write(myVar)", waiting
for cat to read from its input pipe, while cat blocks at a write
to its output stream, waiting for you to start reading and freeing
up buffer space. Pipe loops are tricky business.

Popular solutions are to make either the input or output stream
a disk file, or to create another thread (or process) to be an
active reader or writer.


--
--Bryan
 
Reply With Quote
 
Bryan Olson
Guest
Posts: n/a
 
      03-13-2008
I wrote:
> [...] Pipe loops are tricky business.
>
> Popular solutions are to make either the input or output stream
> a disk file, or to create another thread (or process) to be an
> active reader or writer.


Or asynchronous I/O. On Unix-like systems, you can select() on
the underlying file descriptors. (MS-Windows async mechanisms are
not as well exposed by the Python standard library.)


--
--Bryan
 
Reply With Quote
 
Benjamin Watine
Guest
Posts: n/a
 
      03-14-2008
Bryan Olson a écrit :
> I wrote:
>> [...] Pipe loops are tricky business.
>>
>> Popular solutions are to make either the input or output stream
>> a disk file, or to create another thread (or process) to be an
>> active reader or writer.

>
> Or asynchronous I/O. On Unix-like systems, you can select() on
> the underlying file descriptors. (MS-Windows async mechanisms are
> not as well exposed by the Python standard library.)
>


Hi Bryan

Thank you so much for your advice. You're right, I just made a test with
a 10 MB input stream, and it hangs exactly like you said (on
cat.stdin.write(myStdin))...

I don't want to use disk files. In reality, this script was previously
done in bash using disk files, but I had problems with that solution
(the files wasn't always cleared, and sometimes, I've found a part of
previous input at the end of the next input.)

That's why I want to use python, just to not use disk files.

Could you give me more information / examples about the two solutions
you've proposed (thread or asynchronous I/O) ?

Thank you !

Ben
 
Reply With Quote
 
Floris Bruynooghe
Guest
Posts: n/a
 
      03-14-2008
On Mar 14, 11:37 am, Benjamin Watine <(E-Mail Removed)> wrote:
> Bryan Olson a écrit :
>
> > I wrote:
> >> [...] Pipe loops are tricky business.

>
> >> Popular solutions are to make either the input or output stream
> >> a disk file, or to create another thread (or process) to be an
> >> active reader or writer.

>
> > Or asynchronous I/O. On Unix-like systems, you can select() on
> > the underlying file descriptors. (MS-Windows async mechanisms are
> > not as well exposed by the Python standard library.)

>
> Hi Bryan
>
> Thank you so much for your advice. You're right, I just made a test with
> a 10 MB input stream, and it hangs exactly like you said (on
> cat.stdin.write(myStdin))...
>
> I don't want to use disk files. In reality, this script was previously
> done in bash using disk files, but I had problems with that solution
> (the files wasn't always cleared, and sometimes, I've found a part of
> previous input at the end of the next input.)
>
> That's why I want to use python, just to not use disk files.
>
> Could you give me more information / examples about the two solutions
> you've proposed (thread or asynchronous I/O) ?


The source code of the subprocess module shows how to do it with
select IIRC. Look at the implementation of the communicate() method.

Regards
Floris
 
Reply With Quote
 
bryanjugglercryptographer@yahoo.com
Guest
Posts: n/a
 
      03-14-2008
Floris Bruynooghe wrote:
> Benjamin Watine wrote:
> > Could you give me more information / examples about the two solutions
> > you've proposed (thread or asynchronous I/O) ?

>
> The source code of the subprocess module shows how to do it with
> select IIRC. Look at the implementation of the communicate() method.


And here's a thread example, based on Benjamin's code:

import subprocess
import thread

def readtobox(pipe, box):
box.append(pipe.read())

cat = subprocess.Popen('cat', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE)

myVar = str(range(1000000)) # arbitrary test data.

box = []
thread.start_new_thread(readtobox, (cat.stdout, box))
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()
myNewVar = box[0]

assert myNewVar == myVar
print len(myNewVar), "bytes piped around."


--
--Bryan

 
Reply With Quote
 
bryanjugglercryptographer@yahoo.com
Guest
Posts: n/a
 
      03-14-2008
I wrote:
> And here's a thread example, based on Benjamin's code:

[...]

Doh! Race condition. Make that:

import subprocess
import thread
import Queue

def readtoq(pipe, q):
q.put(pipe.read())

cat = subprocess.Popen('cat', shell=True, stdin=subprocess.PIPE,
stdout=subprocess.PIPE)

myVar = str(range(1000000)) # arbitrary test data.

q = Queue.Queue()
thread.start_new_thread(readtoq, (cat.stdout, q))
cat.stdin.write(myVar)
cat.stdin.close()
cat.wait()
myNewVar = q.get()

assert myNewVar == myVar
print len(myNewVar), "bytes piped around."


--
--Bryan



 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
Difference between Session["var"] and Session("var") thomson ASP .Net 10 06-20-2005 01:02 PM
Difference between Session["var"] and Session("var") thomson ASP .Net 0 06-20-2005 10:54 AM
Threads.. Session var lost, App var ok Alvin Bruney ASP .Net 1 12-02-2003 01:56 AM
does "struct_name var = { 0 }; " fill var with 0? Fred C++ 3 08-10-2003 09:44 AM



Advertisments