Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   Re: Retrieving the full command line (http://www.velocityreviews.com/forums/t956791-re-retrieving-the-full-command-line.html)

 Tim Golden 01-22-2013 03:07 PM

Re: Retrieving the full command line

On 22/01/2013 14:53, Terry Reedy wrote:
> On 1/22/2013 4:24 AM, Tim Golden wrote:
>> [Python 2.7/3.3 (and hg tip) running on Windows. Not Windows-specific,
>> though].
>>
>> I use the python -mpackage incantation to run a package which has a
>> __main__.py module and which uses relative imports internally.
>>
>> I'm developing under cherrypy which includes a reloader for development.
>> The reloader attempts to rebuild the original
>> command line by combining sys.executable and sys.argv and then does an
>> execv.
>>
>> There does not appear to be any way within Python of determining the
>> command line I used. The combination of sys.executable and sys.argv in
>> this case will look like: "c:\python33\python.exe app/__main__.py". But
>> running this precludes the use of package-relative imports.

>
> If I understand right, the reloader should be updated to translate
> 'x/__main__.py' to '-m x'. Filenames of form'__x__' are reserved, in a
> sense, like similar identifiers in programs, and '__main__.py' should
> not be used for a file meant to executed directly.

To be clear: it's Python itself, not the reloader, which is coming up
with __main__.py. sys.executable is "c:\python33\python.exe" and
sys.argv is ['c:\path\to\__main__.py'] for a program which has been
started by "c:\python33\python.exe -mpath\to".

Obviously, there is any number of ways around this specific issue,
including what you suggest: a canonical rewrite of "python
path\to\__main__.py" into "python -mpath\to". But it's not clear to me
that this rewrite should be the responsibility of calling code.

TJG

 Steven D'Aprano 01-22-2013 11:46 PM

Re: Retrieving the full command line

On Tue, 22 Jan 2013 15:07:18 +0000, Tim Golden wrote:

> On 22/01/2013 14:53, Terry Reedy wrote:
>> On 1/22/2013 4:24 AM, Tim Golden wrote:
>>> [Python 2.7/3.3 (and hg tip) running on Windows. Not Windows-specific,
>>> though].
>>>
>>> I use the python -mpackage incantation to run a package which has a
>>> __main__.py module and which uses relative imports internally.
>>>
>>> I'm developing under cherrypy which includes a reloader for
>>> development. The reloader attempts to rebuild the original command
>>> line by combining sys.executable and sys.argv and then does an execv.
>>>
>>> There does not appear to be any way within Python of determining the
>>> command line I used. The combination of sys.executable and sys.argv in
>>> this case will look like: "c:\python33\python.exe app/__main__.py".
>>> But running this precludes the use of package-relative imports.

>>
>> If I understand right, the reloader should be updated to translate
>> 'x/__main__.py' to '-m x'. Filenames of form'__x__' are reserved, in a
>> sense, like similar identifiers in programs, and '__main__.py' should
>> not be used for a file meant to executed directly.

>
> To be clear: it's Python itself, not the reloader, which is coming up
> with __main__.py. sys.executable is "c:\python33\python.exe" and
> sys.argv is ['c:\path\to\__main__.py'] for a program which has been
> started by "c:\python33\python.exe -mpath\to".

I don't believe you can give direct paths to the -m flag. It uses the
normal import mechanism to locate a module or package, so you have to
give it a name which would be importable.

c:\python33\python.exe -m app

would work, where "app" is either a package or module:

C:\something\on\PYTHONPATH\app\__main__.py
C:\something\on\PYTHONPATH\app.py

> Obviously, there is any number of ways around this specific issue,
> including what you suggest: a canonical rewrite of "python
> path\to\__main__.py" into "python -mpath\to". But it's not clear to me
> that this rewrite should be the responsibility of calling code.

I am a bit disturbed that you cannot distinguish between:

python C:\something\on\pythonpath\app\__main__.py

python -m app

by inspecting the command line. I consider it a bug, or at least a
misfeature, if Python transforms the command line before making it
available in sys.argv.

--
Steven

 Oscar Benjamin 01-23-2013 12:53 AM

Re: Retrieving the full command line

On 22 January 2013 23:46, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
[SNIP]
>
> I am a bit disturbed that you cannot distinguish between:
>
> python C:\something\on\pythonpath\app\__main__.py
>
> python -m app
>
>
> by inspecting the command line. I consider it a bug, or at least a
> misfeature, if Python transforms the command line before making it
> available in sys.argv.

The purpose of the -m option is that you can run a script that is
located via the Python import path instead of an explicit file path.
The idea is that if '/path/to/somewhere' is in sys.path then:
python -m script arg1 arg2
is equivalent to
python /path/to/somewhere/script.py arg1 arg2

If Python didn't modify sys.argv then 'script.py' would need to be
rewritten to understand that sys.argv would be in a different format
when it was invoked using the -m option.

I believe that it has previously been proposed to include something
like sys.raw_argv, although perhaps for different reasons. Although
something that might be more useful to the OP (and that I at one point
wanted) would be a function similar to execfile but that would launch
a separate process using the same interpreter as the current process.

Oscar

 Steven D'Aprano 01-23-2013 03:58 AM

Re: Retrieving the full command line

On Wed, 23 Jan 2013 00:53:21 +0000, Oscar Benjamin wrote:

> On 22 January 2013 23:46, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote: [SNIP]
>>
>> I am a bit disturbed that you cannot distinguish between:
>>
>> python C:\something\on\pythonpath\app\__main__.py
>>
>> python -m app
>>
>>
>> by inspecting the command line. I consider it a bug, or at least a
>> misfeature, if Python transforms the command line before making it
>> available in sys.argv.

>
> The purpose of the -m option is that you can run a script that is
> located via the Python import path instead of an explicit file path. The
> idea is that if '/path/to/somewhere' is in sys.path then:
> python -m script arg1 arg2
> is equivalent to
> python /path/to/somewhere/script.py arg1 arg2
>
> If Python didn't modify sys.argv then 'script.py' would need to be
> rewritten to understand that sys.argv would be in a different format
> when it was invoked using the -m option.

I don't think that it would be in a different format. Normally people
only care about sys.argv[1:], the actual arguments. argv[0], the name of
the script, already comes in multiple formats: absolute or relative paths.

Currently, if I have a package __main__.py that prints sys.argv, I get
results like this:

steve@runes:~$python3.3 /home/steve/python/testpackage/__main__.py ham spam eggs ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs'] which is correct, that's what I gave on the command line. But: steve@runes:~$ python3.3 -m testpackage ham spam eggs
['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']

The second example is lying. It should say:

['-m testpackage', 'ham', 'spam', 'eggs']

If you are one of the few people who care about argv[0], then you are
already dealing with the fact that the name of the executable script is
not always an absolute path and therefore can vary greatly from one call
to another. Hell, if you are on a system with soft links, the name of the
script in the command line is not even necessarily the name of the
module. So there's not much more effort involved in dealing with one
extra case:

# assuming you care, which most people don't
if sys.argv[0].startswith('-m'):
do_something()
elif os.path.isabs(sys.argv[0]):
do_this()
else: # relative path
do_that()

--
Steven

 Tim Golden 01-23-2013 09:58 AM

Re: Retrieving the full command line

On 23/01/2013 03:58, Steven D'Aprano wrote:
> Currently, if I have a package __main__.py that prints sys.argv, I get
> results like this:
>
> steve@runes:~$python3.3 /home/steve/python/testpackage/__main__.py ham > spam eggs > ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs'] > > > which is correct, that's what I gave on the command line. But: > > steve@runes:~$ python3.3 -m testpackage ham spam eggs
> ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
>
>
> The second example is lying. It should say:
>
> ['-m testpackage', 'ham', 'spam', 'eggs']

Thanks for the input, Steven & Oscar.

Apologies for the confusion over my initial example. Of course, -m runs
something on sys.path, not something in the filesystem as such. I
confused myself because, running on Windows where the current directory
is on the path, I sit in c:\path\to and do python -mapp

Now I look harder, this discussion is basically issue14208:

http://bugs.python.org/issue14208

so I'll probably go and contribute over there.

TJG

 Oscar Benjamin 01-23-2013 10:01 AM

Re: Retrieving the full command line

On 23 January 2013 03:58, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> On Wed, 23 Jan 2013 00:53:21 +0000, Oscar Benjamin wrote:
>
>> On 22 January 2013 23:46, Steven D'Aprano
>> <steve+comp.lang.python@pearwood.info> wrote: [SNIP]
>>>

>> The purpose of the -m option is that you can run a script that is
>> located via the Python import path instead of an explicit file path. The
>> idea is that if '/path/to/somewhere' is in sys.path then:
>> python -m script arg1 arg2
>> is equivalent to
>> python /path/to/somewhere/script.py arg1 arg2
>>
>> If Python didn't modify sys.argv then 'script.py' would need to be
>> rewritten to understand that sys.argv would be in a different format
>> when it was invoked using the -m option.

>
> I don't think that it would be in a different format. Normally people
> only care about sys.argv[1:], the actual arguments. argv[0], the name of
> the script, already comes in multiple formats: absolute or relative paths.
>
> Currently, if I have a package __main__.py that prints sys.argv, I get
> results like this:
>
> steve@runes:~$python3.3 /home/steve/python/testpackage/__main__.py ham > spam eggs > ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs'] > > > which is correct, that's what I gave on the command line. But: > > steve@runes:~$ python3.3 -m testpackage ham spam eggs
> ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
>
>
> The second example is lying. It should say:
>
> ['-m testpackage', 'ham', 'spam', 'eggs']

I don't know why you would expect this. I imagined that you would want

['-m', 'testpackage', 'ham', 'spam', 'eggs']

If the two were combined into one string I would expect it to at least
be a valid argument list:

['-mtestpackage', 'ham', 'spam', 'eggs']

>
>
> If you are one of the few people who care about argv[0], then you are
> already dealing with the fact that the name of the executable script is
> not always an absolute path and therefore can vary greatly from one call
> to another. Hell, if you are on a system with soft links, the name of the
> script in the command line is not even necessarily the name of the
> module. So there's not much more effort involved in dealing with one
> extra case:

Unless I've missed something sys.argv[0] is always a valid path to the
script. Whether it is absolute or not shouldn't matter. For imported
modules the path is available from __name__. For a script that is
executed rather than imported __name__ == "__main__" but the path is
accessible from sys.argv[0]. If you are one of those people who cares
about sys.argv[0] then this is probably the value that you wanted it
to contain.

If it were important for sys.argv to show how exactly the script was
located and executed, then why not also include the 'python3.3'
command line argument (the real argv[0])? sys.argv emulates the argv
that e.g. a C program would get. The real command line used is not
exactly the same since a Python script is not a directly executable
binary, so Python processes the argument list before passing it
through.

In the OP's case, the script is never invoked without -m so the following works:

cmdline = [sys.executable, '-m', __package__] + sys.argv[1:]

In the more general case it would be better to have an API
specifically for this purpose.

Oscar

 Steven D'Aprano 01-24-2013 04:49 AM

Re: Retrieving the full command line

On Wed, 23 Jan 2013 10:01:24 +0000, Oscar Benjamin wrote:

> On 23 January 2013 03:58, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
>> On Wed, 23 Jan 2013 00:53:21 +0000, Oscar Benjamin wrote:
>>
>>> On 22 January 2013 23:46, Steven D'Aprano
>>> <steve+comp.lang.python@pearwood.info> wrote: [SNIP]
>>>>
>>> The purpose of the -m option is that you can run a script that is
>>> located via the Python import path instead of an explicit file path.
>>> The idea is that if '/path/to/somewhere' is in sys.path then:
>>> python -m script arg1 arg2
>>> is equivalent to
>>> python /path/to/somewhere/script.py arg1 arg2
>>>
>>> If Python didn't modify sys.argv then 'script.py' would need to be
>>> rewritten to understand that sys.argv would be in a different format
>>> when it was invoked using the -m option.

>>
>> I don't think that it would be in a different format. Normally people
>> only care about sys.argv[1:], the actual arguments. argv[0], the name
>> of the script, already comes in multiple formats: absolute or relative
>> paths.
>>
>> Currently, if I have a package __main__.py that prints sys.argv, I get
>> results like this:
>>
>> steve@runes:~$python3.3 /home/steve/python/testpackage/__main__.py ham >> spam eggs >> ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs'] >> >> >> which is correct, that's what I gave on the command line. But: >> >> steve@runes:~$ python3.3 -m testpackage ham spam eggs
>> ['/home/steve/python/testpackage/__main__.py', 'ham', 'spam', 'eggs']
>>
>>
>> The second example is lying. It should say:
>>
>> ['-m testpackage', 'ham', 'spam', 'eggs']

>
> I don't know why you would expect this. I imagined that you would want
>
> ['-m', 'testpackage', 'ham', 'spam', 'eggs']

No. argv[0] is intended to be the script being called, argv[1:] for the
arguments to the script. Given the two choices:

1) Break every Python script that expects argv[1:] to be the arguments
to the script, forcing them to decide whether they should look at
argv[1:] or argv[2:] according to whether or not argv[0] == '-m';

or

2) don't break anything, but make a very small addition to the semantics
of argv[0] (was: "the path to the script", add "or -m and the name of
module/package") that won't break anyone's code;

there's practically no choice in the matter.

> If the two were combined into one string I would expect it to at least
> be a valid argument list:
>
> ['-mtestpackage', 'ham', 'spam', 'eggs']

Okay, fair point. I didn't consider that.

Note however that there is an ambiguity between calling "python -mspam"
and calling a script literally named "-mspam". But that same ambiguity
exists in the shell, so I don't consider it a problem. You cannot call a
script named -mspam unless you use something like this "python ./-mspam".

>> If you are one of the few people who care about argv[0], then you are
>> already dealing with the fact that the name of the executable script is
>> not always an absolute path and therefore can vary greatly from one
>> call to another. Hell, if you are on a system with soft links, the name
>> of the script in the command line is not even necessarily the name of
>> the module. So there's not much more effort involved in dealing with
>> one extra case:

>
> Unless I've missed something sys.argv[0] is always a valid path to the
> script. Whether it is absolute or not shouldn't matter.

Sure. But if you care about argv[0] (say, you want to pull out the name
of the script at runtime, instead of hard-coding it), then you need to be
aware that you could be given an absolute path, a relative path, a bare
script name, or the path of a softlink to the file you actually care
about. Adding one more trivially simple case is not a large burden.

People hardly ever care about argv[0]. At least, I don't think I ever
have. But the OP does, and Python mangling argv[0] is causing him grief
because it lies, claiming to have called the __main__.py of his package
directly when in fact he called it with -m.

> For imported
> modules the path is available from __name__. For a script that is
> executed rather than imported __name__ == "__main__" but the path is
> accessible from sys.argv[0]. If you are one of those people who cares
> about sys.argv[0] then this is probably the value that you wanted it to
> contain.

I'm wary about guessing what people "probably" want, and therefore lying
about what they actually got. That's DWIM coding, and that almost always
ends in tears.

> If it were important for sys.argv to show how exactly the script was
> located and executed, then why not also include the 'python3.3' command
> line argument (the real argv[0])? sys.argv emulates the argv that e.g. a
> C program would get. The real command line used is not exactly the same
> since a Python script is not a directly executable binary, so Python
> processes the argument list before passing it through.

Also a good point. To some degree, we're constrained by backwards
compatibility -- there's only so much change we can do without breaking
code, and setting argv[0] to the python executable instead of the script
is too big a change.

In any case, you can get that information using sys.executable, or at
least you can get the path of the actual Python binary, or you can use
sys.version (or equivalent) to determine which version of Python you're
using.

This does mean that you can't play dirty hacks like some C binaries do,
where they change their behaviour depending on whether you call them via
one path or another path. vi does that. But that's hardly a big loss.

Contrariwise, I don't believe that there is currently *any* way to
distinguish between running a script with or without -m. That should be
fixed.

--
Steven

 Chris Angelico 01-24-2013 05:06 AM

Re: Retrieving the full command line

On Thu, Jan 24, 2013 at 3:49 PM, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
> Note however that there is an ambiguity between calling "python -mspam"
> and calling a script literally named "-mspam". But that same ambiguity
> exists in the shell, so I don't consider it a problem. You cannot call a
> script named -mspam unless you use something like this "python ./-mspam".

Another spanner for your works: "python -- -mspam" succeeds. That sets
argv[0] to '-mspam'.

> People hardly ever care about argv[0]. At least, I don't think I ever
> have. But the OP does, and Python mangling argv[0] is causing him grief
> because it lies, claiming to have called the __main__.py of his package
> directly when in fact he called it with -m.

Usually when I reference argv[0], $0, or any equivalent, it's for a usage display - eg: USAGE:$0 [opts] infile [outfile]
--foo Fooify the file
--bar Burn your computer to the ground

So I don't particularly care about symlinks or relative paths (if it
worked once, it'll probably work another time). But ambiguities may be
an issue.

ChrisA

 Oscar Benjamin 01-24-2013 10:06 AM

Re: Retrieving the full command line

On 24 January 2013 04:49, Steven D'Aprano
<steve+comp.lang.python@pearwood.info> wrote:
[SNIP]
>
> Contrariwise, I don't believe that there is currently *any* way to
> distinguish between running a script with or without -m. That should be
> fixed.

As I said earlier in the thread, the __package__ module global
distinguishes the two cases:

~$mkdir pkg ~$ touch pkg/__init__.py
~$vim pkg/__main__.py ~$ cat pkg/__main__.py
import sys
if __package__ is None:
cmdline = [sys.executable] + sys.argv
else:
cmdline = [sys.executable, '-m', __package__] + sys.argv[1:]
print(cmdline)
~$python pkg/__main__.py arg1 arg2 ['q:\\tools\\Python27\\python.exe', 'pkg/__main__.py', 'arg1', 'arg2'] ~$ python -m pkg arg1 arg2
['q:\\tools\\Python27\\python.exe', '-m', 'pkg', 'arg1', 'arg2']

Oscar

 Tim Golden 01-24-2013 10:56 AM

Re: Retrieving the full command line

On 24/01/2013 10:06, Oscar Benjamin wrote:
> On 24 January 2013 04:49, Steven D'Aprano
> <steve+comp.lang.python@pearwood.info> wrote:
> [SNIP]
>>
>> Contrariwise, I don't believe that there is currently *any* way to
>> distinguish between running a script with or without -m. That should be
>> fixed.

>
> As I said earlier in the thread, the __package__ module global
> distinguishes the two cases:
>
> ~$mkdir pkg > ~$ touch pkg/__init__.py
> ~$vim pkg/__main__.py > ~$ cat pkg/__main__.py
> import sys
> if __package__ is None:
> cmdline = [sys.executable] + sys.argv
> else:
> cmdline = [sys.executable, '-m', __package__] + sys.argv[1:]
> print(cmdline)
> ~$python pkg/__main__.py arg1 arg2 > ['q:\\tools\\Python27\\python.exe', 'pkg/__main__.py', 'arg1', 'arg2'] > ~$ python -m pkg arg1 arg2
> ['q:\\tools\\Python27\\python.exe', '-m', 'pkg', 'arg1', 'arg2']

Reasonable (and thanks for the clear example), but it doesn't work
if the package which is reconstructing the command line the package
which was the target of the original command line. In my case,
I'm making use of the cherrypy reloader, whose __package__ is
cherrypy.process. But the command which invoked the program was
python -m myapp.

ie I'm issuing "python -m myapp". In myapp.__main__ I'm importing
cherrypy, itself a package, and somewhere in cherrypy.whatever there is
code which attempts to reconstruct the command line.

TJG

All times are GMT. The time now is 07:29 PM.