Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > PyWart: Import resolution order

Reply
Thread Tools

PyWart: Import resolution order

 
 
Rick Johnson
Guest
Posts: n/a
 
      01-11-2013

Python's import resolution order is terrible.[1]

The fact that Python looks in the stdlib _first_ is not a good idea. It would seem more intuitive for a custom "math" module (living in the current directory) to /override/ the stlib "math" module. The proper order is as follows:

1. Current package or directory
2. stdlib
3. under the bed
4. who cares at this point

Look, if you want to keep you foolish imports starting from the stdlib, fine, just create a switch so we can change it to "package/directory" if we like.

If /only/ the Python gods had placed all stdlib modules in a package named,i don't know, "lib" then none of this would be an issue. You could happilyhave a stdlib::math and a mylib::math, and when you import either module your code will reflect that path explicitly. Hmm, this last sentence reminded me of something??? Oh yes -> "Explicit is better than implicit".


[1]: Yes, yes, i know about sys.path.insert. *puke*!
 
Reply With Quote
 
 
 
 
Chris Angelico
Guest
Posts: n/a
 
      01-11-2013
On Fri, Jan 11, 2013 at 5:13 PM, Rick Johnson
<(E-Mail Removed)> wrote:
> The fact that Python looks in the stdlib _first_ is not a good idea. It would seem more intuitive for a custom "math" module (living in the current directory) to /override/ the stlib "math" module. The proper order is as follows:
>
> 1. Current package or directory
> 2. stdlib
> 3. under the bed
> 4. who cares at this point


Why is it better to import from the current directory first? Windows
has that policy for executable commands; Unix, on the other hand,
requires that you put an explicit path for anything that isn't in the
standard search path. Which of these options is the more likely to
produce security holes and/or unexpected behaviour?

Welcome back to the list, Rick. Got any demonstrable code for Python 4000 yet?

ChrisA
 
Reply With Quote
 
 
 
 
Terry Reedy
Guest
Posts: n/a
 
      01-11-2013
On 1/11/2013 1:13 AM, Rick Johnson wrote:
>
> Python's import resolution order is terrible.[1]
>
> The fact that Python looks in the stdlib _first_ is not a good idea.


And the fact is that it does not do so. The order depends on sys.path,
and '' is the first entry.

> It would seem more intuitive for a custom "math" module (living in
> the current directory) to /override/ the stlib "math" module.


Try it.

This is a nuisance though, when not intended. Someone writes a
random.py, and a year later in the same directory, an unrelated
hopscotch.py, which tries to import random.exponential. The import fails
and they post here, having forgotten about their own random.py, which
does not have such a function. Posts like this happen a few times a year.

--
Terry Jan Reedy

 
Reply With Quote
 
Michael Torrie
Guest
Posts: n/a
 
      01-11-2013
On 01/10/2013 11:13 PM, Rick Johnson wrote:
>
> Python's import resolution order is terrible.[1]
>
> The fact that Python looks in the stdlib _first_ is not a good idea.


Whether or not the default behavior is desirable or not, sys.path is set
by default to look in the current directory first on any Python
distribution I have looked at. As Terry says, this fact causes a lot of
problems for newbies who accidentally override standard library modules
with their own python files and chaos ensues.

If your python installation does not search the current directory first,
then you must have changed sys.path.
 
Reply With Quote
 
Rick Johnson
Guest
Posts: n/a
 
      01-12-2013
On Friday, January 11, 2013 7:35:37 AM UTC-6, Terry Reedy wrote:
> On 1/11/2013 1:13 AM, Rick Johnson wrote:
> > The fact that Python looks in the stdlib _first_ is not a good idea.

>
> And the fact is that it does not do so. The order depends on sys.path,
> and '' is the first entry.
>
> > It would seem more intuitive for a custom "math" module (living in
> > the current directory) to /override/ the stlib "math" module.

>
> This is a nuisance though, when not intended. Someone writes a
> random.py, and a year later in the same directory, an unrelated
> hopscotch.py, which tries to import random.exponential. The import fails
> and they post here, having forgotten about their own random.py, which
> does not have such a function. Posts like this happen a few times a year.


That's why i also mentioned the failure of Python to wrap stdlib modules in a package. If we would protect all built-in modules by placing them in a package (lib or py) then this problem would never happen.

Of course many people will **** and moan about the extra typing. I say, you have a choice: a few extra chars or multitudes of extra headaches -- I choose the first option.

Since more time is spent /maintaining/ code bases than /writing/ them, the explicit path is always the correct path to choose. Anyone who says otherwise is either careless or selfish (aka: seeking job security).
 
Reply With Quote
 
Rick Johnson
Guest
Posts: n/a
 
      01-12-2013
On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
> Why is it better to import from the current directory first?


Opps. I was not explicit enough with my explanation . I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


But the real solution is not to change the default resolution order. The real solution is to wrap all builtin modules into a package and use full paths to access every module. But wait, i have an even better idea... read below!

> Windows
> has that policy for executable commands; Unix, on the other hand,
> requires that you put an explicit path for anything that isn't in the
> standard search path. Which of these options is the more likely to
> produce security holes and/or unexpected behaviour?


I prefer the latter of course .

I think if python where *strict* about full paths for non-builtins, then wewould be in a better place. But there is an even better solution! Force all python users to wrap THEIR modules in a toplevel package. Maybe even create the package for them. YES!. Call it "lib". Any "naked" modules (meaning modules that are not in a package) would have to be accessed starting from"lib". Of course professionals like you and i are already using packages to properly nest out modules, but the newbie's won't realize the power of packaging modules for some time, so create the default "lib" package for them..

For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".

So the conclusion is:

* We encourage python programmers to use packages so they avoid any name clashes with built-in modules.

* if they refuse fine, any "naked" modules they create will be packaged ina default package (call it "lib", "userlib", or whatever) and will requirethem to prefix the module name with "lib." -- or "lib:" if i get my way!

By doing this we solve the many problems related to module name resolution orders and we create cleaner code bases. Damn i am full of good ideas!

> Welcome back to the list, Rick. Got any demonstrable code
> for Python 4000 yet?


I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.
 
Reply With Quote
 
Rick Johnson
Guest
Posts: n/a
 
      01-12-2013
On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
> Why is it better to import from the current directory first?


Opps. I was not explicit enough with my explanation . I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


But the real solution is not to change the default resolution order. The real solution is to wrap all builtin modules into a package and use full paths to access every module. But wait, i have an even better idea... read below!

> Windows
> has that policy for executable commands; Unix, on the other hand,
> requires that you put an explicit path for anything that isn't in the
> standard search path. Which of these options is the more likely to
> produce security holes and/or unexpected behaviour?


I prefer the latter of course .

I think if python where *strict* about full paths for non-builtins, then wewould be in a better place. But there is an even better solution! Force all python users to wrap THEIR modules in a toplevel package. Maybe even create the package for them. YES!. Call it "lib". Any "naked" modules (meaning modules that are not in a package) would have to be accessed starting from"lib". Of course professionals like you and i are already using packages to properly nest out modules, but the newbie's won't realize the power of packaging modules for some time, so create the default "lib" package for them..

For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".

So the conclusion is:

* We encourage python programmers to use packages so they avoid any name clashes with built-in modules.

* if they refuse fine, any "naked" modules they create will be packaged ina default package (call it "lib", "userlib", or whatever) and will requirethem to prefix the module name with "lib." -- or "lib:" if i get my way!

By doing this we solve the many problems related to module name resolution orders and we create cleaner code bases. Damn i am full of good ideas!

> Welcome back to the list, Rick. Got any demonstrable code
> for Python 4000 yet?


I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.
 
Reply With Quote
 
Chris Angelico
Guest
Posts: n/a
 
      01-12-2013
On Sat, Jan 12, 2013 at 4:28 PM, Rick Johnson
<(E-Mail Removed)> wrote:
> On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
>> Welcome back to the list, Rick. Got any demonstrable code
>> for Python 4000 yet?

>
> I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.


So all I have to do is paper my programming world and I win?

Sorry, can't hang around arguing about module search paths and your
recommendations to add piles of syntactic salt. Gotta finish building
and playing with today's Linux kernel (3.2.7, it's looking good so
far).

ChrisA
 
Reply With Quote
 
Ian Kelly
Guest
Posts: n/a
 
      01-12-2013
On Fri, Jan 11, 2013 at 10:28 PM, Rick Johnson
<(E-Mail Removed)> wrote:
> On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
>> Why is it better to import from the current directory first?

>
> Opps. I was not explicit enough with my explanation . I meant, "look inthe current directory FIRST when in a package". Since many times (most alltimes) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


And again, in Python 2.x this is already the case. When importing in
a package, it tries to do a relative import before it even looks at
sys.path.

> I think if python where *strict* about full paths for non-builtins, then we would be in a better place.


And again, in Python 3, where implicit relative imports have been
removed from the language, it already is strict about using full
paths. You can still do relative imports, but you have to be explicit
about them.

> For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".


What if I create a package named "math"? Does that also automatically
get renamed to "lib.math"? How is it decided what package names are
proper; is it just because it happens to clash with a stdlib name that
the package gets magically renamed?

What if I create a package, and then later a module with the same name
happens to be added to the stdlib? My program that uses the package
just breaks because it no longer imports the correct thing?

> Damn i am full of good ideas!


Your ideas might be better if you first spent some time gaining a
better understanding of how the language works as is.
 
Reply With Quote
 
alex23
Guest
Posts: n/a
 
      01-12-2013
On 12 Jan, 14:50, Rick Johnson <(E-Mail Removed)> wrote:
> Of course many people will **** and moan about the extra typing.


You just ignored the fact that your original claim was incorrect and
kept going on with your rant anyway.

> Since more time is spent /maintaining/ code bases than /writing/ them


In your case, more time is actually spent on insisting _other_ people
maintain code bases.

Everyone: PLEASE STOP FEEDING THE TROLL
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
import order or cross import Roland Hedberg Python 5 01-03-2007 11:25 PM
LCD TV resolution / DVD resolution ? slonkak DVD Video 0 11-13-2006 02:34 PM
How do you change the Modelsim Cursor Resolution (not simulation resolution) Andrew FPGA VHDL 0 09-26-2005 04:05 AM
Resolution resolution Simon Digital Photography 4 02-27-2004 01:53 PM
ISO Resolution Chart and Printing Resolution Jack Yeazel Digital Photography 0 08-11-2003 11:19 PM



Advertisments