Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Python as a scripting language. Alternative to bash script?

Reply
Thread Tools

Re: Python as a scripting language. Alternative to bash script?

 
 
Michael Torrie
Guest
Posts: n/a
 
      06-28-2010
On 06/28/2010 05:48 AM, Dave Pawson wrote:
> Main queries are: Ease of calling out to bash to use something like
> imageMagick or Java? Ease of grabbing return parameters? E.g. convert
> can return both height and width of an image. Can this be returned to
> the Python program? Can Python access the exit status of a program?


Sure. I've created a module called runcmd that does 90% of what I want
(easy access to stdout, stderr, error code). I've attached it to this
e-mail. Feel free to use it; this post puts my code into the public domain.

> I'd prefer the advantages of using Python, just wondering if I got so
> far with the port then found it wouldn't do something?


Python really isn't a shell scripting language. So there are things
that Bash does much better, such as spawning processes and piping them
together. I've tried over the years to create a pythonic library that
would let me do that, but haven't found a good syntax that I like.

It turns out, though, that much of what I use piping for in Bash is to
run external processes to do things that I could use python modules for.
For example, I typically pipe stuff to cut a lot to get certain fields.
For example:

ps ax | cut -c1-5

In python I could simply take the output of "ps ax" and use python's
own, superior, cutting routines (using my module):

(err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
for x in stdout.split('\n'):
print x.strip().split()[0]

Sure it's a couple more lines in this case, but in other cases, python's
abilities make it simpler than bash. A great document on how you can
exploit python's abilities (particularly generators) to replace bash
pipelines is here: http://www.dabeaz.com/generators/





 
Reply With Quote
 
 
 
 
sturlamolden
Guest
Posts: n/a
 
      07-06-2010
On 28 Jun, 19:39, Michael Torrie <torr...@gmail.com> wrote:

> In python I could simply take the output of "ps ax" and use python's
> own, superior, cutting routines (using my module):
>
> (err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
> for x in stdout.split('\n'):
> * * print x.strip().split()[0]


Or you just pass the stdout of one command as stdin to another. That
is equivalent of piping with bash.

 
Reply With Quote
 
 
 
 
Michael Torrie
Guest
Posts: n/a
 
      07-06-2010
On 07/06/2010 04:12 AM, sturlamolden wrote:
> On 28 Jun, 19:39, Michael Torrie <torr...@gmail.com> wrote:
>
>> In python I could simply take the output of "ps ax" and use python's
>> own, superior, cutting routines (using my module):
>>
>> (err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
>> for x in stdout.split('\n'):
>> print x.strip().split()[0]

>
> Or you just pass the stdout of one command as stdin to another. That
> is equivalent of piping with bash.


Consider this contrived example:

tail -f /var/log/messages | grep openvpn

While it's possible to set up pipes and spawn programs in parallel to
operate on the pipes, in practice it's simpler to tell subprocess.Popen
to use a shell and then just rely on Bash's very nice syntax for setting
up the pipeline. Then just read the final output in python. If you set
the stdout descriptor to non-blocking, you could read output as it came.
 
Reply With Quote
 
member thudfoo
Guest
Posts: n/a
 
      07-06-2010
On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie <> wrote:
> On 07/06/2010 04:12 AM, sturlamolden wrote:
>> On 28 Jun, 19:39, Michael Torrie <torr...@gmail.com> wrote:
>>
>>> In python I could simply take the output of "ps ax" and use python's
>>> own, superior, cutting routines (using my module):
>>>
>>> (err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
>>> for x in stdout.split('\n'):
>>> * * print x.strip().split()[0]

>>
>> Or you just pass the stdout of one command as stdin to another. That
>> is equivalent of piping with bash.

>
> Consider this contrived example:
>
> tail -f /var/log/messages | grep openvpn
>
> While it's possible to set up pipes and spawn programs in parallel to
> operate on the pipes, in practice it's simpler to tell subprocess.Popen
> to use a shell and then just rely on Bash's very nice syntax for setting
> up the pipeline. *Then just read the final output in python. *If you set
> the stdout descriptor to non-blocking, you could read output as it came.
> --
> http://mail.python.org/mailman/listinfo/python-list
>


Is this a discussion about the pipes module in the std library?
 
Reply With Quote
 
Chris Rebert
Guest
Posts: n/a
 
      07-07-2010
On Tue, Jul 6, 2010 at 1:35 PM, member thudfoo <> wrote:
> On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie <> wrote:
>> On 07/06/2010 04:12 AM, sturlamolden wrote:
>>> On 28 Jun, 19:39, Michael Torrie <torr...@gmail.com> wrote:
>>>> In python I could simply take the output of "ps ax" and use python's
>>>> own, superior, cutting routines (using my module):
>>>>
>>>> (err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
>>>> for x in stdout.split('\n'):
>>>> Â* Â* print x.strip().split()[0]
>>>
>>> Or you just pass the stdout of one command as stdin to another. That
>>> is equivalent of piping with bash.

>>
>> Consider this contrived example:
>>
>> tail -f /var/log/messages | grep openvpn
>>
>> While it's possible to set up pipes and spawn programs in parallel to
>> operate on the pipes, in practice it's simpler to tell subprocess.Popen
>> to use a shell and then just rely on Bash's very nice syntax for setting
>> up the pipeline. Â*Then just read the final output in python. Â*If you set
>> the stdout descriptor to non-blocking, you could read output as it came.

>
> Is this a discussion about the pipes module in the std library?


No, though that module is not irrelevant to Mr. Torrie's argument.

Cheers,
Chris
--
http://blog.rebertia.com
 
Reply With Quote
 
Chris Rebert
Guest
Posts: n/a
 
      07-07-2010
On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie <> wrote:
> On 07/06/2010 04:12 AM, sturlamolden wrote:
>> On 28 Jun, 19:39, Michael Torrie <torr...@gmail.com> wrote:
>>> In python I could simply take the output of "ps ax" and use python's
>>> own, superior, cutting routines (using my module):
>>>
>>> (err, stdout, stderr) = runcmd.run( [ 'ps', 'ax' ] )
>>> for x in stdout.split('\n'):
>>> Â* Â* print x.strip().split()[0]

>>
>> Or you just pass the stdout of one command as stdin to another. That
>> is equivalent of piping with bash.

>
> Consider this contrived example:
>
> tail -f /var/log/messages | grep openvpn
>
> While it's possible to set up pipes and spawn programs in parallel to
> operate on the pipes, in practice it's simpler to tell subprocess.Popen
> to use a shell and then just rely on Bash's very nice syntax for setting
> up the pipeline.


Until there's a Python variable involved that is, unless you want to
overlook all the edge cases or do the escaping all by yourself (and
then pray you did it right).

Cheers,
Chris
--
http://blog.rebertia.com
 
Reply With Quote
 
Michael Torrie
Guest
Posts: n/a
 
      07-07-2010
On 07/06/2010 09:34 PM, Chris Rebert wrote:
> On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie <> wrote:
>> While it's possible to set up pipes and spawn programs in parallel to
>> operate on the pipes, in practice it's simpler to tell subprocess.Popen
>> to use a shell and then just rely on Bash's very nice syntax for setting
>> up the pipeline.

>
> Until there's a Python variable involved that is, unless you want to
> overlook all the edge cases or do the escaping all by yourself (and
> then pray you did it right).


Very good point. This is a problem that the pipes module suffers from
as well.

Although we learned in the other thread on escaping SQL statements that
escaping is faster, easier and just as safe as other parameterization
mechanisms. Uh huh.

Back on target, a library similar to pipes that was safe (pipes is not)
and had a pythonic syntax would be cool. pipes module works alright,
syntax wise, but it's not a clean syntax.
 
Reply With Quote
 
Chris Rebert
Guest
Posts: n/a
 
      07-07-2010
On Wed, Jul 7, 2010 at 8:31 AM, Michael Torrie <> wrote:
> On 07/06/2010 09:34 PM, Chris Rebert wrote:
>> On Tue, Jul 6, 2010 at 6:40 AM, Michael Torrie <> wrote:
>>> While it's possible to set up pipes and spawn programs in parallel to
>>> operate on the pipes, in practice it's simpler to tell subprocess.Popen
>>> to use a shell and then just rely on Bash's very nice syntax for setting
>>> up the pipeline.

>>
>> Until there's a Python variable involved that is, unless you want to
>> overlook all the edge cases or do the escaping all by yourself (and
>> then pray you did it right).

>
> Very good point. Â*This is a problem that the pipes module suffers from
> as well.
>
> Although we learned in the other thread on escaping SQL statements that
> escaping is faster, easier and just as safe as other parameterization
> mechanisms. Â*Uh huh.
>
> Back on target, a library similar to pipes that was safe (pipes is not)
> and had a pythonic syntax would be cool. Â*pipes module works alright,
> syntax wise, but it's not a clean syntax.


Actually, your original post inspired me to take a crack at writing
something like that yesterday:
http://rebertia.com/code/subproc_pipelines.py

Thoughts anyone? (Written on a whim, so no tests or docs at present.)

Cheers,
Chris
--
http://blog.rebertia.com
 
Reply With Quote
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      07-24-2010
In message <mailman.301.1278423632.1673.python->, Michael
Torrie wrote:

> While it's possible to set up pipes and spawn programs in parallel to
> operate on the pipes, in practice it's simpler to tell subprocess.Popen
> to use a shell and then just rely on Bash's very nice syntax for setting
> up the pipeline.


Just be careful about properly escaping any special characters in the file
names. See my other thread about escaping data, where some respondents have
expressed a strong allergy to any form of embedding one language inside
another.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Python as a scripting language. Alternative to bash script? Benjamin Kaplan Python 13 07-07-2010 07:19 AM
Re: Python as a scripting language. Alternative to bash script? Jorgen Grahn Python 0 06-29-2010 09:18 AM
Guide to using python for bash-style scripting 4zumanga Python 4 05-24-2006 12:02 AM
(patch for Bash) GTK+2 + Bash William Park XML 3 07-14-2005 12:09 AM
Python is the best and most popular general purpose scripting language; the universal scripting language Ron Stephens Python 23 04-12-2004 05:32 PM



Advertisments
 



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57