Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Popen Question

Reply
Thread Tools

Popen Question

 
 
moogyd
Guest
Posts: n/a
 
      11-04-2010
Hi,
I usually use csh for my simulation control scripts, but these scripts
are becoming more complex, so I plan to use python for the next
project.
To this end, I am looking at subprocess.Popen() to actually call the
simulations, and have a very basic question which is demonstrated
below.

[sde:staff@lbux03 ~]$ python
Python 2.6 (r26:66714, Feb 21 2009, 02:16:04)
[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, subprocess
>>> os.environ['MYVAR'] = "myval"
>>> p = subprocess.Popen(['echo', '$MYVAR'],shell=True)
>>>
>>> p = subprocess.Popen(['echo', '$MYVAR'])
>>> $MYVAR


>>> p = subprocess.Popen('echo $MYVAR',shell=True)
>>> myval


>>> p = subprocess.Popen('echo $MYVAR')

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1106, in
_execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

I am not really sure I understand these results.
1) No idea what is going on
2) As (1). What isn't myval printed out (rather than $MYVAR)
3) Works as I wanted it to
4) Why do I need shell=True ?
The documentation isn't very clear to me (it seems you need to
understand the underlying system calls).

Can anyone explain (or provide link) for this behaviour in simple
English?
Thanks,
Steven
 
Reply With Quote
 
 
 
 
Ravi
Guest
Posts: n/a
 
      11-04-2010
On Nov 4, 7:06*pm, moogyd <(E-Mail Removed)> wrote:
> Hi,
> I usually use csh for my simulation control scripts, but these scripts
> are becoming more complex, so I plan to use python for the next
> project.
> To this end, I am looking at subprocess.Popen() to actually call the
> simulations, and have a very basic question which is demonstrated
> below.
>
> [sde:staff@lbux03 ~]$ python
> Python 2.6 (r26:66714, Feb 21 2009, 02:16:04)
> [GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>
> >>> import os, subprocess
> >>> os.environ['MYVAR'] = "myval"
> >>> p = subprocess.Popen(['echo', '$MYVAR'],shell=True)

>
> >>> p = subprocess.Popen(['echo', '$MYVAR'])
> >>> $MYVAR
> >>> p = subprocess.Popen('echo $MYVAR',shell=True)
> >>> myval
> >>> p = subprocess.Popen('echo $MYVAR')

>
> Traceback (most recent call last):
> * File "<stdin>", line 1, in <module>
> * File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
> * * errread, errwrite)
> * File "/usr/lib64/python2.6/subprocess.py", line 1106, in
> _execute_child
> * * raise child_exception
> OSError: [Errno 2] No such file or directory
>
> I am not really sure I understand these results.
> 1) No idea what is going on
> 2) As (1). What isn't myval printed out (rather than $MYVAR)
> 3) Works as I wanted it to
> 4) Why do I need shell=True ?
> The documentation isn't very clear to me (it seems you need to
> understand the underlying system calls).
>
> Can anyone explain (or provide link) for this behaviour in simple
> English?
> Thanks,
> Steven


try giving /bin/echo
 
Reply With Quote
 
 
 
 
Alain Ketterlin
Guest
Posts: n/a
 
      11-04-2010
moogyd <(E-Mail Removed)> writes:

>>>> import os, subprocess
>>>> os.environ['MYVAR'] = "myval"
>>>> p = subprocess.Popen(['echo', '$MYVAR'],shell=True)
>>>>
>>>> p = subprocess.Popen(['echo', '$MYVAR'])
>>>> $MYVAR

>
>>>> p = subprocess.Popen('echo $MYVAR',shell=True)
>>>> myval

>
>>>> p = subprocess.Popen('echo $MYVAR')

> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
> errread, errwrite)
> File "/usr/lib64/python2.6/subprocess.py", line 1106, in
> _execute_child
> raise child_exception
> OSError: [Errno 2] No such file or directory
>
> I am not really sure I understand these results.
> 1) No idea what is going on
> 2) As (1). What isn't myval printed out (rather than $MYVAR)
> 3) Works as I wanted it to
> 4) Why do I need shell=True ?


Expanding $MYVAR into its value is a feature of the shell (afaik all
shells use the same syntax). Popen without shell=True uses the execvp()
system call directly, without going through the shell variable expansion
process (cases 2 and 4 above). For example, case 4 above asks execvp to
(find and) execute a program named "echo $MYVAR" (an 11-letter name,
where the fifth letter is space and the sixth is $ -- a perfectly valid
file/program name).

Then, if you use shell=True with a list, only the first word is used as
a command, and the others are kept in positional parameters. That's why
your first try fails (try 'sh -c echo $HOME' in a shell, without the
single quotes, and you'll get empty output).

> The documentation isn't very clear to me (it seems you need to
> understand the underlying system calls).


You're probably right. The base fact here is: the use of variables is a
feature of the shell. No shell, no variable.

> Can anyone explain (or provide link) for this behaviour in simple
> English?


Shell variables are explained in detail in any shell man page. The
execvp() system call has its own man page.

-- Alain.
 
Reply With Quote
 
Chris Torek
Guest
Posts: n/a
 
      11-05-2010
In article <(E-Mail Removed)>
moogyd <(E-Mail Removed)> wrote:
>[sde:staff@lbux03 ~]$ python
>Python 2.6 (r26:66714, Feb 21 2009, 02:16:04)
>[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
>Type "help", "copyright", "credits" or "license" for more information.
>>>> import os, subprocess
>>>> os.environ['MYVAR'] = "myval"
>>>> p = subprocess.Popen(['echo', '$MYVAR'],shell=True)


Alain Ketterlin has already explained these to some extent.
Here is a bit more.

This runs, underneath:

['/bin/sh', '-c', 'echo', '$MYVAR']

(with arguments expressed as a Python list). /bin/sh takes the
string after '-c' as a command, and the remaining argument(s) if
any are assigned to positional parameters ($0, $1, etc).

If you replace the command with something a little more explicit,
you can see this:

>>> p = subprocess.Popen(

... [r'echo \$0=$0 \$1=$1', 'arg0', '$MYVAR'], shell=True)
>>> $0=arg0 $1=$MYVAR

p.wait()
0
>>>


(I like to call p.communicate() or p.wait(), although p.communicate()
is pretty much a no-op if you have not done any redirecting. Note that
p.communicate() does a p.wait() for you.)

>>>> p = subprocess.Popen(['echo', '$MYVAR'])
>>>> $MYVAR


This time, as Alain noted, the shell does not get involved so no
variable expansion occurs. However, you could do it yourself:

>>> p = subprocess.Popen(['echo', os.environ['MYVAR']])
>>> myval

p.wait()
0
>>>


>>>> p = subprocess.Popen('echo $MYVAR',shell=True)
>>>> myval


(here /bin/sh does the expansion, because you invoked it)

>>>> p = subprocess.Popen('echo $MYVAR')

>Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/lib64/python2.6/subprocess.py", line 595, in __init__
> errread, errwrite)
> File "/usr/lib64/python2.6/subprocess.py", line 1106, in
>_execute_child
> raise child_exception
>OSError: [Errno 2] No such file or directory


This attempted to run the executable named 'echo $MYVAR'. It did
not exist so the underlying exec (after the fork) failed. The
exception was passed back to the subprocess module, which raised
it in the parent for you to see.

If you were to create an executable named 'echo $MYVAR' (including
the blank and dollar sign) somewhere in your path (or use an explicit
path to it), it would run. I will also capture the actual output
this time:

$ cat '/tmp/echo $MYVAR'
#! /usr/bin/awk NR>1{print}
this is a self-printing file
anything after the first line has NR > 1, so gets printed
$ chmod +x '/tmp/echo $MYVAR'
$ python
Python 2.5.1 (r251:54863, Feb 6 2009, 19:02:12)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> p = subprocess.Popen('/tmp/echo $MYVAR', stdout=subprocess.PIPE)
>>> print p.communicate()[0]

this is a self-printing file
anything after the first line has NR > 1, so gets printed

>>> p.returncode

0
>>>


Incidentally, fun with #!: you can make self-renaming scripts:

sh-3.2$ echo '#! /bin/mv' > /tmp/selfmove; chmod +x /tmp/selfmove
sh-3.2$ ls /tmp/*move*
/tmp/selfmove
sh-3.2$ /tmp/selfmove /tmp/I_moved
sh-3.2$ ls /tmp/*move*
/tmp/I_moved
sh-3.2$

or even self-removing scripts:

sh-3.2$ echo '#! /bin/rm' > /tmp/rmme; chmod +x /tmp/rmme
sh-3.2$ /tmp/rmme
sh-3.2$ /tmp/rmme
sh: /tmp/rmme: No such file or directory

(nothing to do with python, just the way #! interpreter lines work).
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) +1 801 277 2603
email: gmail (figure it out) http://web.torek.net/torek/index.html
 
Reply With Quote
 
moogyd
Guest
Posts: n/a
 
      11-07-2010
Hi,

Thanks everyone for the replies - it is now clearer.

Steven

 
Reply With Quote
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      11-08-2010
In message <(E-Mail Removed)>, Chris Torek wrote:

> ['/bin/sh', '-c', 'echo', '$MYVAR']
>
> (with arguments expressed as a Python list). /bin/sh takes the
> string after '-c' as a command, and the remaining argument(s) if
> any are assigned to positional parameters ($0, $1, etc).


Doesn’t work. I don’t know what happens to the extra arguments, but they
just seem to be ignored if -c is specified.

sh -c 'echo hi'

echoes “hi”, while

sh -c echo hi

just outputs a blank line.
 
Reply With Quote
 
Mark Wooding
Guest
Posts: n/a
 
      11-08-2010
Lawrence D'Oliveiro <(E-Mail Removed)_zealand> writes:

> In message <(E-Mail Removed)>, Chris Torek wrote:
>
> > ['/bin/sh', '-c', 'echo', '$MYVAR']
> >
> > (with arguments expressed as a Python list). /bin/sh takes the
> > string after '-c' as a command, and the remaining argument(s) if
> > any are assigned to positional parameters ($0, $1, etc).

>
> Doesn’t work.


What doesn't work? You were being given an explanation, not a solution.

> I don’t know what happens to the extra arguments, but they just seem
> to be ignored if -c is specified.


The argument to -c is taken as a shell script; the remaining arguments
are made available as positional parameters to the script as usual (only
starting with $0 rather than $1, for some unknown reason).

-- [mdw]
 
Reply With Quote
 
Ian
Guest
Posts: n/a
 
      11-08-2010
On Nov 8, 2:43*am, (E-Mail Removed) (Mark Wooding) wrote:
> > I don’t know what happens to the extra arguments, but they just seem
> > to be ignored if -c is specified.

>
> The argument to -c is taken as a shell script; the remaining arguments
> are made available as positional parameters to the script as usual (only
> starting with $0 rather than $1, for some unknown reason).


Perhaps this example better demonstrates what is going on:

>>> p = subprocess.Popen(['echo one $0 three $1 five', 'two', 'four'],

.... shell=True)
one two three four five

Cheers,
Ian
 
Reply With Quote
 
Hans Mulder
Guest
Posts: n/a
 
      11-08-2010
Ian wrote:
> On Nov 8, 2:43 am, (E-Mail Removed) (Mark Wooding) wrote:
>>> I don’t know what happens to the extra arguments, but they just seem
>>> to be ignored if -c is specified.

>> The argument to -c is taken as a shell script; the remaining arguments
>> are made available as positional parameters to the script as usual (only
>> starting with $0 rather than $1, for some unknown reason).

>
> Perhaps this example better demonstrates what is going on:
>
>>>> p = subprocess.Popen(['echo one $0 three $1 five', 'two', 'four'],

> ... shell=True)
> one two three four five


Maybe I'm thick, but I still don't understand. If I run a shell script,
then the name of the script is in $0 and the first positional arguments
is in $1, similar to how Python sets up sys.argv.

But in this case the first positional argument is in $0. Why is that?

Puzzled,

-- HansM


 
Reply With Quote
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      11-08-2010
In message <4cd87b24$0$81481$(E-Mail Removed)4all.nl>, Hans Mulder wrote:

> But in this case the first positional argument is in $0.


That’s what confused me.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Question about pipes/os.popen Kevin Walzer Python 4 09-15-2006 08:42 PM
File.popen/IO.popen hariwise@gmail.com Ruby 1 05-20-2006 08:20 AM
Python or 4NT? With a question or two about popen() Robin Siebler Python 6 09-13-2004 08:55 PM
Re: simple popen question Jim Benson Python 1 04-03-2004 10:39 AM
simple popen question Jim Benson Python 1 04-03-2004 01:52 AM



Advertisments