Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Idiom for running compiled python scripts?

Reply
Thread Tools

Idiom for running compiled python scripts?

 
 
Mark
Guest
Posts: n/a
 
      03-21-2007
Hi, I'm new to python and looking for a better idiom to use for the
manner I have been organising my python scripts. I've googled all over
the place about this but found absolutely nothing.

I'm a linux/unix command line guy quite experienced in shell scripts
etc. I have a heap of command line utility scripts which I run directly.
What is the best way to create python command line scripts but exploit
the (loadonly) speed-up benefit of python compiled code?

E.g. say I have a python script "myprog.py". I could just execute that
directly each time but that means it is "compiled" each time I run it
which is not efficient and adds to startup time. I have been creating a
complimentary script "myprog" stub which just does:

#!/usr/bin/env python
from myprog import main
if __name__ == "__main__":
main()

Of course this compiles myprog.py into myprog.pyc on first run as I am
wanting.

I have one of these stubs for all my python scripts I've created so far.
Is there not a better way? Do I have to create a separate stub each
time? I find it a bit messy to require a pair of scripts for each
utility and it also contributes some inefficiency. Given the above stub
is so boilerplate, why does python not provide a general stub/utility
mechanism for this?
 
Reply With Quote
 
 
 
 
Bjoern Schliessmann
Guest
Posts: n/a
 
      03-21-2007
Mark wrote:

> E.g. say I have a python script "myprog.py". I could just execute
> that directly each time but that means it is "compiled" each time
> I run it which is not efficient and adds to startup time.


Did you measure the performance hit in your case?

> I have one of these stubs for all my python scripts I've created
> so far. Is there not a better way? Do I have to create a separate
> stub each time? I find it a bit messy to require a pair of scripts
> for each utility and it also contributes some inefficiency. Given
> the above stub is so boilerplate, why does python not provide a
> general stub/utility mechanism for this?


I've noticed that calling the interpreter with pre-compiled pyc
files also works.

Regards,


Björn


--
BOFH excuse #68:

only available on a need to know basis

 
Reply With Quote
 
 
 
 
Max Erickson
Guest
Posts: n/a
 
      03-21-2007
Mark <(E-Mail Removed)> wrote:

....
> #!/usr/bin/env python
> from myprog import main
> if __name__ == "__main__":
> main()
>
> Of course this compiles myprog.py into myprog.pyc on first run as
> I am wanting.
>
> I have one of these stubs for all my python scripts I've created
> so far. Is there not a better way? Do I have to create a separate
> stub each time? I find it a bit messy to require a pair of
> scripts for each utility and it also contributes some
> inefficiency. Given the above stub is so boilerplate, why does
> python not provide a general stub/utility mechanism for this?


I don't know of a better way to organize things, but as an
alternative, you could have a script where you import each of the
scripts that you want compiled, python will write the compiled files
to disk when you run it(thus avoiding the need to split the other
scripts). There are also the py_compile and compileall modules, which
have facilities for generating byte code.

More here:

http://effbot.org/zone/python-compile.htm

under 'Compiling python modules to byte code'.


max

 
Reply With Quote
 
Mark
Guest
Posts: n/a
 
      03-23-2007
So given the lack of response it seems that there is probably no such
idiom and that I should not be concerned by the inefficiency inherent in
running .py scripts directly?

I did some time tests and sure, the speed gain is slight, but it is a
gain none the less.
 
Reply With Quote
 
Steve Holden
Guest
Posts: n/a
 
      03-23-2007
Mark wrote:
> So given the lack of response it seems that there is probably no such
> idiom and that I should not be concerned by the inefficiency inherent in
> running .py scripts directly?
>
> I did some time tests and sure, the speed gain is slight, but it is a
> gain none the less.


Someone already told you - compile the files manually (see the compile
built-in function) and then use

python file.pyc

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Recent Ramblings http://holdenweb.blogspot.com

 
Reply With Quote
 
Steve Holden
Guest
Posts: n/a
 
      03-23-2007
Mark wrote:
> So given the lack of response it seems that there is probably no such
> idiom and that I should not be concerned by the inefficiency inherent in
> running .py scripts directly?
>
> I did some time tests and sure, the speed gain is slight, but it is a
> gain none the less.


Sorry, what you really need is the compileFile() function from the
compiler module.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Recent Ramblings http://holdenweb.blogspot.com

 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      03-23-2007
On Fri, 23 Mar 2007 01:01:15 +0000, Mark wrote:

> So given the lack of response it seems that there is probably no such
> idiom and that I should not be concerned by the inefficiency inherent in
> running .py scripts directly?
>
> I did some time tests and sure, the speed gain is slight, but it is a
> gain none the less.


Since you've done these tests already, perhaps you can tell us what gain
you actually got?

Here's a test I did:

[steve@apple ~]$ time python script.py
the result is 166166000

real 0m0.555s
user 0m0.470s
sys 0m0.011s
[steve@apple ~]$ time python script.pyc
the result is 166166000

real 0m0.540s
user 0m0.456s
sys 0m0.011s


That gives me an absolute gain of 15ms which is a percentage gain of about
3%. But don't forget the time it takes you to type the extra "c" at the
end of the script, even with filename completion. The average human
reaction time is something between 200 and 270 milliseconds, so unless
you're saving at least 200ms, typing that extra "c" at the end actually
wastes time.

Of course you have to type the "c". You're not deleting the source files
away are you? *wink*



--
Steven D'Aprano

 
Reply With Quote
 
Mark
Guest
Posts: n/a
 
      03-23-2007
On Fri, 23 Mar 2007 14:03:12 +1100, Steven D'Aprano wrote:
> Since you've done these tests already, perhaps you can tell us what gain
> you actually got?


About the same as you, ~20 msecs for my small script samples.

> Of course you have to type the "c". You're not deleting the source files
> away are you? *wink*


Sorry, the wink is lost on me?

Of course I am not deleting the sources. In fact, I am also talking
about python scripts being called from shell scripts. I guess I'm just
surprised that the python installation does not provide a small stub
invoker, e.g:

A small script called "python_compile_and_run" in "pseudo" code:

#!/usr/bin/env python
import sys

# Following is invalid syntax unfortunately
from sys.argv[1].rstrip('.py') import main

sys.argv = sys.argv[1:]
if __name__ == "__main__":
main()

so I could just do a "python_compile_and_run myscript.py" and it would
do what I want, i.e. run myscript.pyc if available and valid, generate
and run it if necessary.
 
Reply With Quote
 
Gerard Flanagan
Guest
Posts: n/a
 
      03-23-2007
On Mar 23, 8:30 am, Mark <(E-Mail Removed)> wrote:
>
> Of course I am not deleting the sources. In fact, I am also talking
> about python scripts being called from shell scripts.


There's a nice recipe in Python Cookbook (Martelli et al.) for this.
It involves zipping your .pyc files and adding a shell stub. Never
used it before but I'm going to need something similar in the near
future, probably with a text templating system such as Cheetah
(www.cheetahtemplate.org).

HTH

Gerard



 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      03-23-2007
On Fri, 23 Mar 2007 07:30:58 +0000, Mark wrote:

> On Fri, 23 Mar 2007 14:03:12 +1100, Steven D'Aprano wrote:
>> Since you've done these tests already, perhaps you can tell us what gain
>> you actually got?

>
> About the same as you, ~20 msecs for my small script samples.


Well, I think that pretty much answers your question about whether it is
worth pre-compiling short shell scripts: you save about 20ms in execution
time, and lose 200ms in typing time. (Maybe a bit less if you are a
fast typist and don't use auto-completion.) You do the maths.


>> Of course you have to type the "c". You're not deleting the source files
>> away are you? *wink*

>
> Sorry, the wink is lost on me?


It is because I didn't really think you were deleting the source files.
That would be incredibly stupid. But I mentioned it just in case some
not-so-bright spark decided to argue that you could use auto-completion
without needing to type that final "c" if you deleted the source file.

Presumably now somebody is going to suggest merely *moving* the source
files into another directory, thus spending a minute or two each time they
edit a script re-arranging files in order to save twenty or thirty
milliseconds when they execute the script. Hey, if your time is so
valuable that 20ms means that much to you, go for it.


> Of course I am not deleting the sources. In fact, I am also talking
> about python scripts being called from shell scripts. I guess I'm just
> surprised that the python installation does not provide a small stub
> invoker, e.g:
>
> A small script called "python_compile_and_run" in "pseudo" code:


[snip pseudo-code]

> so I could just do a "python_compile_and_run myscript.py" and it would
> do what I want, i.e. run myscript.pyc if available and valid, generate
> and run it if necessary.


You shouldn't expect Python to come with every imaginable special-purpose
script already written for you! Besides, it's pretty simple to get that
functionality by hand when you need it, or automatically for that matter.

Here's one (untested) script that executes the pyc file in a subshell if
it exists and is new enough, and compiles it if it doesn't.


import os, sys, compiler
from stat import ST_MTIME as MT
if __name__ == "__main__":
scriptname = sys.argv[1]
compiledname = scriptname + "c"
if not os.path.exists(compiledname) or \
os.stat(compiledname)[MT] < os.stat(scriptname)[MT]:
# compiled file doesn't exist, or is too old
compiler.compileFile(scriptname)
assert os.path.exists(compiledname)
resultcode = os.system('python %s' % compiledname)
sys.exit(resultcode)

Now don't forget to test whether launching the subshell takes longer than
the 20ms you might save. All that effort, and wouldn't it be ironic if it
was actually *slower* than executing the script from scratch each time...


--
Steven.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Running compiled windows service python script Aspersieman Python 1 05-13-2008 01:19 PM
If I create a page, then it's compiled upon first request, where cani find the compiled code?? lander ASP .Net 5 03-05-2008 04:34 PM
Running compiled Python files Shankar Python 0 05-03-2006 12:42 PM
g++ compiled C++ code called from gcc compiled C code Klaus Schneider C++ 1 12-02-2004 01:44 PM
ASP function run differently when running compiled from DLL Peter Row ASP .Net 0 01-26-2004 11:48 AM



Advertisments