Velocity Reviews > Re: Retrieving the full command line

# Re: Retrieving the full command line

Tim Golden
Guest
Posts: n/a

 01-24-2013
On 24/01/2013 10:56, Tim Golden wrote:
> if the package which is reconstructing the command line the package
> which was the target of the original command line.

Sorry:

if the package which is reconstructing the command line *is not*
the package which was the target of the original command line.

TJG

Oscar Benjamin
Guest
Posts: n/a

 01-24-2013
On 24 January 2013 10:56, Tim Golden <(E-Mail Removed)> wrote:
> On 24/01/2013 10:06, Oscar Benjamin wrote:
>> On 24 January 2013 04:49, Steven D'Aprano
>> <(E-Mail Removed)> wrote:
>> [SNIP]
>>>
>>> Contrariwise, I don't believe that there is currently *any* way to
>>> distinguish between running a script with or without -m. That should be
>>> fixed.

>>
>> As I said earlier in the thread, the __package__ module global
>> distinguishes the two cases:
>>
>> ~$mkdir pkg >> ~$ touch pkg/__init__.py
>> ~$vim pkg/__main__.py >> ~$ cat pkg/__main__.py
>> import sys
>> if __package__ is None:
>> cmdline = [sys.executable] + sys.argv
>> else:
>> cmdline = [sys.executable, '-m', __package__] + sys.argv[1:]
>> print(cmdline)
>> ~$python pkg/__main__.py arg1 arg2 >> ['q:\\tools\\Python27\\python.exe', 'pkg/__main__.py', 'arg1', 'arg2'] >> ~$ python -m pkg arg1 arg2
>> ['q:\\tools\\Python27\\python.exe', '-m', 'pkg', 'arg1', 'arg2']

>
> Reasonable (and thanks for the clear example), but it doesn't work
> if the package which is reconstructing the command line the package
> which was the target of the original command line. In my case,
> I'm making use of the cherrypy reloader, whose __package__ is
> cherrypy.process. But the command which invoked the program was
> python -m myapp.
>
> ie I'm issuing "python -m myapp". In myapp.__main__ I'm importing
> cherrypy, itself a package, and somewhere in cherrypy.whatever there is
> code which attempts to reconstruct the command line.

Easy enough:

~$mkdir pkg ~$ touch pkg/__init__.py
~$vim pkg/__main__.py ~$ cat pkg/__main__.py
import pkg.whatever
~$vim pkg/whatever.py ~$ cat pkg/whatever.py
import sys
import pkg.__main__ as main
cmdline = [sys.executable, '-m', main.__package__] + sys.argv[1:]
print(cmdline)
~$python -m pkg ['q:\\tools\\Python27\\python.exe', '-m', 'pkg'] ~$ python -m pkg arg1 arg32
['q:\\tools\\Python27\\python.exe', '-m', 'pkg', 'arg1', 'arg32']

I don't really understand what your spec is. Why do you need to
inspect this information from sys.argv? Can you not just always use
'python -m pkg' as your entry point?

Oscar

Tim Golden
Guest
Posts: n/a

 01-24-2013
On 24/01/2013 11:30, Oscar Benjamin wrote:
> I don't really understand what your spec is. Why do you need to
> inspect this information from sys.argv? Can you not just always use
> 'python -m pkg' as your entry point?

Sorry about the confusion. I think my original point was simply one
of surprise that sys.argv wouldn't essentially mirror the elements
of the command line which I used to get there.
The specifics of my use-case weren't really too important.

attempts to restart (via execv) whatever process was responsible for
loading it in the first place, via an identical or equivalent command
line. The current (cherrypy) code simply joins sys.executable and
sys.argv but this fails in the face of python -m as we have seen.

The cherrypy package has no especial knowledge of the structure of the
application which imported it and so must piece together the command
line somehow. Clearly, I can take various approaches of the sort
which you've outlined, or subclass the reloader, or fetch the original
command line from the OS, etc. It's not that this is a showstopper,
merely slightly surprising. (To me).

TJG

Oscar Benjamin
Guest
Posts: n/a

 01-24-2013
On 24 January 2013 13:45, Tim Golden <(E-Mail Removed)> wrote:
> On 24/01/2013 11:30, Oscar Benjamin wrote:
>> I don't really understand what your spec is. Why do you need to
>> inspect this information from sys.argv? Can you not just always use
>> 'python -m pkg' as your entry point?

>

[SNIP]
>
> attempts to restart (via execv) whatever process was responsible for
> loading it in the first place, via an identical or equivalent command
> line. The current (cherrypy) code simply joins sys.executable and
> sys.argv but this fails in the face of python -m as we have seen.
>
> The cherrypy package has no especial knowledge of the structure of the
> application which imported it and so must piece together the command
> line somehow. Clearly, I can take various approaches of the sort
> which you've outlined, or subclass the reloader, or fetch the original
> command line from the OS, etc. It's not that this is a showstopper,
> merely slightly surprising. (To me).

Ok I understand. Then I guess you want:

import __main__
pkg = __main__.__package__

Oscar

Tim Golden
Guest
Posts: n/a

 01-24-2013
On 24/01/2013 15:28, Oscar Benjamin wrote:
> On 24 January 2013 13:45, Tim Golden <(E-Mail Removed)> wrote:
>> On 24/01/2013 11:30, Oscar Benjamin wrote:
>>> I don't really understand what your spec is. Why do you need to
>>> inspect this information from sys.argv? Can you not just always use
>>> 'python -m pkg' as your entry point?

>>

> [SNIP]
>>
>> attempts to restart (via execv) whatever process was responsible for
>> loading it in the first place, via an identical or equivalent command
>> line. The current (cherrypy) code simply joins sys.executable and
>> sys.argv but this fails in the face of python -m as we have seen.
>>
>> The cherrypy package has no especial knowledge of the structure of the
>> application which imported it and so must piece together the command
>> line somehow. Clearly, I can take various approaches of the sort
>> which you've outlined, or subclass the reloader, or fetch the original
>> command line from the OS, etc. It's not that this is a showstopper,
>> merely slightly surprising. (To me).

>
> Ok I understand. Then I guess you want:
>
> import __main__
> pkg = __main__.__package__

Brilliant. Never thought of importing __main__. Thanks.

For the benefit of anyone still watching, the code (which has to be
compatible back to 2.3) looks something like this:

<code>
import __main__

# [.. .snip ...]

try:
is_package = bool(__main__.__package__)
except NameError:
is_package = False
if is_package:
args = [sys.executable, '-m', __main__.__package__] + sys.argv[1:]
else:
args = [sys.executable] + sys.argv

os.chdir(_startup_cwd) # avoids relative/absolute issues
os.execv(args[0], args)

</code>

I don't pretend it's foolproot, but it certainly works for my particular
case. Nor have I considered it against all the cases identified in PEP
432: http://www.python.org/dev/peps/pep-0...uring-sys-argv

TJG

Oscar Benjamin
Guest
Posts: n/a

 01-24-2013
On 24 January 2013 15:51, Tim Golden <(E-Mail Removed)> wrote:
> On 24/01/2013 15:28, Oscar Benjamin wrote:
>> On 24 January 2013 13:45, Tim Golden <(E-Mail Removed)> wrote:
>>> On 24/01/2013 11:30, Oscar Benjamin wrote:
>>>> I don't really understand what your spec is. Why do you need to
>>>> inspect this information from sys.argv? Can you not just always use
>>>> 'python -m pkg' as your entry point?
>>>

>> [SNIP]
>>>
>>> attempts to restart (via execv) whatever process was responsible for
>>> loading it in the first place, via an identical or equivalent command
>>> line. The current (cherrypy) code simply joins sys.executable and
>>> sys.argv but this fails in the face of python -m as we have seen.
>>>
>>> The cherrypy package has no especial knowledge of the structure of the
>>> application which imported it and so must piece together the command
>>> line somehow. Clearly, I can take various approaches of the sort
>>> which you've outlined, or subclass the reloader, or fetch the original
>>> command line from the OS, etc. It's not that this is a showstopper,
>>> merely slightly surprising. (To me).

>>
>> Ok I understand. Then I guess you want:
>>
>> import __main__
>> pkg = __main__.__package__

>
> Brilliant. Never thought of importing __main__. Thanks.
>
> For the benefit of anyone still watching, the code (which has to be
> compatible back to 2.3) looks something like this:
>
> <code>
> import __main__
>
> # [.. .snip ...]
>
> try:
> is_package = bool(__main__.__package__)
> except NameError:
> is_package = False
> if is_package:
> args = [sys.executable, '-m', __main__.__package__] + sys.argv[1:]
> else:
> args = [sys.executable] + sys.argv
>
> os.chdir(_startup_cwd) # avoids relative/absolute issues
> os.execv(args[0], args)
>
> </code>
>
> I don't pretend it's foolproot, but it certainly works for my particular
> case. Nor have I considered it against all the cases identified in PEP
> 432: http://www.python.org/dev/peps/pep-0...uring-sys-argv

Does it work if you use the -m option to run a module rather than a script?

Oscar

Oscar Benjamin
Guest
Posts: n/a

 01-24-2013
On 24 January 2013 16:08, Oscar Benjamin <(E-Mail Removed)> wrote:
> On 24 January 2013 15:51, Tim Golden <(E-Mail Removed)> wrote:
>> On 24/01/2013 15:28, Oscar Benjamin wrote:
>>> On 24 January 2013 13:45, Tim Golden <(E-Mail Removed)> wrote:
>>>> On 24/01/2013 11:30, Oscar Benjamin wrote:
>>>>> I don't really understand what your spec is. Why do you need to
>>>>> inspect this information from sys.argv? Can you not just always use
>>>>> 'python -m pkg' as your entry point?
>>>>
>>> [SNIP]
>>>>
>>>> attempts to restart (via execv) whatever process was responsible for
>>>> loading it in the first place, via an identical or equivalent command
>>>> line. The current (cherrypy) code simply joins sys.executable and
>>>> sys.argv but this fails in the face of python -m as we have seen.
>>>>
>>>> The cherrypy package has no especial knowledge of the structure of the
>>>> application which imported it and so must piece together the command
>>>> line somehow. Clearly, I can take various approaches of the sort
>>>> which you've outlined, or subclass the reloader, or fetch the original
>>>> command line from the OS, etc. It's not that this is a showstopper,
>>>> merely slightly surprising. (To me).
>>>
>>> Ok I understand. Then I guess you want:
>>>
>>> import __main__
>>> pkg = __main__.__package__

>>
>> Brilliant. Never thought of importing __main__. Thanks.
>>
>> For the benefit of anyone still watching, the code (which has to be
>> compatible back to 2.3) looks something like this:
>>
>> <code>
>> import __main__
>>
>> # [.. .snip ...]
>>
>> try:
>> is_package = bool(__main__.__package__)
>> except NameError:
>> is_package = False
>> if is_package:
>> args = [sys.executable, '-m', __main__.__package__] + sys.argv[1:]
>> else:
>> args = [sys.executable] + sys.argv
>>
>> os.chdir(_startup_cwd) # avoids relative/absolute issues
>> os.execv(args[0], args)
>>
>> </code>
>>
>> I don't pretend it's foolproot, but it certainly works for my particular
>> case. Nor have I considered it against all the cases identified in PEP
>> 432: http://www.python.org/dev/peps/pep-0...uring-sys-argv

>
> Does it work if you use the -m option to run a module rather than a script?

Sorry that was written incorrectly. I meant to say: does it work when
a module is directly on sys.path rather than as a submodule of a
package? In this case __package__ is set to the empty string if run
with -m or None if run with a direct path. So the check needs to be
"__package__ is not None" rather than "bool(__package__)".

Oscar

Tim Golden
Guest
Posts: n/a

 01-24-2013
On 24/01/2013 16:53, Oscar Benjamin wrote:
>> Does it work if you use the -m option to run a module rather than a script?

>
> Sorry that was written incorrectly. I meant to say: does it work when
> a module is directly on sys.path rather than as a submodule of a
> package? In this case __package__ is set to the empty string if run
> with -m or None if run with a direct path. So the check needs to be
> "__package__ is not None" rather than "bool(__package__)".

The answer is: it depends. Given the code I outlined earlier:

A package-based module run via -m (python -m package.module) works
as described (including the implicit __main__ module, my
primary use-case).

A module run from the filesystem (python c:\path\to\module.py) works
by dropping through through to the not is_package logic branch.

A module run via -m (python -m module) actually works by accident,
because it too drops through to the not is_package branch and is
rerun with its full filesystem path. This doesn't have the same
problems as running a package from the filesystem because relative
imports aren't an issue. I don't know if there are any other differences
between python -mmodule and python c:\path\to\module.py.

As you say, a more refined check could determine a blank __package__
as opposed to a None __package__. But this code (cherrypy) must also
cope with version of Python before 2.6 which didn't even have a
__package__ attribute, muddying the waters that little bit further.

TJG

Oscar Benjamin
Guest
Posts: n/a

 01-24-2013
On 24 January 2013 17:13, Tim Golden <(E-Mail Removed)> wrote:
> On 24/01/2013 16:53, Oscar Benjamin wrote:
>>> Does it work if you use the -m option to run a module rather than a script?

>>
>> Sorry that was written incorrectly. I meant to say: does it work when
>> a module is directly on sys.path rather than as a submodule of a
>> package? In this case __package__ is set to the empty string if run
>> with -m or None if run with a direct path. So the check needs to be
>> "__package__ is not None" rather than "bool(__package__)".

>
> The answer is: it depends. Given the code I outlined earlier:
>
> A package-based module run via -m (python -m package.module) works
> as described (including the implicit __main__ module, my
> primary use-case).

Does it work in the "python -m package.module" case? It looks to me as
if the code below will run "python -m package" instead. I think you
need to split the module name out of sys.argv[0] and put that into the
command line as well.

> import __main__
>
> # [.. .snip ...]
>
> try:
> is_package = bool(__main__.__package__)
> except NameError:
> is_package = False
> if is_package:
> args = [sys.executable, '-m', __main__.__package__] + sys.argv[1:]
> else:
> args = [sys.executable] + sys.argv
>
> os.chdir(_startup_cwd) # avoids relative/absolute issues
> os.execv(args[0], args)

I believe "python -m package" and "python -m package.__main__" are
equivalent so it doesn't matter in that case.

>
> A module run from the filesystem (python c:\path\to\module.py) works
> by dropping through through to the not is_package logic branch.
>
> A module run via -m (python -m module) actually works by accident,
> because it too drops through to the not is_package branch and is
> rerun with its full filesystem path. This doesn't have the same
> problems as running a package from the filesystem because relative
> imports aren't an issue. I don't know if there are any other differences
> between python -mmodule and python c:\path\to\module.py.

There is a (probably pathological) case in which running the script
via a file path and running it via -m are not equivalent. "python
dir/script.py" places "dir" at the top of sys.path. "python -m script"
only works if "dir" is in sys.path but it needn't be at the top. This
means that the order of sys.path is changed and import conflicts may
be resolved differently in the two cases:

~$cat ~/.local/lib/python2.7/site-packages/script.py print("Running script.py") import optparse ~$ cat ~/.local/lib/python2.7/site-packages/optparse.py
raise ImportError('Wrong optparse!')
~$python ~/.local/lib/python2.7/site-packages/script.py Running script.py Traceback (most recent call last): File "/home/oscar/.local/lib/python2.7/site-packages/script.py", line 2, in <module> import optparse File "/home/oscar/.local/lib/python2.7/site-packages/optparse.py", line 1, in <module> raise ImportError('Wrong optparse!') ImportError: Wrong optparse! ~$ python -m script
Running script.py
~$python ~/.local/lib/python2.7/site-packages/script.py Running script.py Traceback (most recent call last): File "/home/oscar/.local/lib/python2.7/site-packages/script.py", line 2, in <module> import optparse File "/home/oscar/.local/lib/python2.7/site-packages/optparse.py", line 1, in <module> raise ImportError('Wrong optparse!') ImportError: Wrong optparse! ~$ python -m script
Running script.py

>
> As you say, a more refined check could determine a blank __package__
> as opposed to a None __package__. But this code (cherrypy) must also
> cope with version of Python before 2.6 which didn't even have a
> __package__ attribute, muddying the waters that little bit further.

Then you won't be able to use this method to get the -m switch to work
on Python < 2.6. If it's ok to just never try the -m switch on those
versions then it's as simple as doing:
pkg = getattr(__main__, '__package__', None)

Oscar

Tim Golden
Guest
Posts: n/a

 01-24-2013
On 24/01/2013 20:01, Oscar Benjamin wrote:
> On 24 January 2013 17:13, Tim Golden <(E-Mail Removed)> wrote:
>> A package-based module run via -m (python -m package.module) works
>> as described (including the implicit __main__ module, my
>> primary use-case).

>
> Does it work in the "python -m package.module" case? It looks to me as
> if the code below will run "python -m package" instead. I think you
> need to split the module name out of sys.argv[0] and put that into the
> command line as well.

Good catch. Unless I am, once again, mistaken, I can't see any way
of recreating a dotted module path which is running as a main module
without the somewhat fragile expedient of walking back up the
filepath and hoping that nothing strange has happened to the imports
on the way down? (pywin32: I'm looking at you!)

And we haven't even touched on zipped archives or other invented
loaders (pulling modules from a database or direct from the internet).

I think, though, that the point which with I started this thread is
still valid: that to reconstruct the command line with which Python
was started, you need an increasingly awkward set of conditions and
checks. As opposed to asking the interpreter: please give me the command
line which started you.

How common that particular requirement actually is, I couldn't say.
But the two cases which clearly could use it are: the reloader
setup we've been discussing here; and the apply-new-version-and-restart
which I'm hoping to use in another setup at work.

Thanks for all the help and input on this, Oscar

TJG