Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   PyWart: Import resolution order (http://www.velocityreviews.com/forums/t956408-pywart-import-resolution-order.html)

Rick Johnson 01-11-2013 06:13 AM

PyWart: Import resolution order
 

Python's import resolution order is terrible.[1]

The fact that Python looks in the stdlib _first_ is not a good idea. It would seem more intuitive for a custom "math" module (living in the current directory) to /override/ the stlib "math" module. The proper order is as follows:

1. Current package or directory
2. stdlib
3. under the bed
4. who cares at this point

Look, if you want to keep you foolish imports starting from the stdlib, fine, just create a switch so we can change it to "package/directory" if we like.

If /only/ the Python gods had placed all stdlib modules in a package named,i don't know, "lib" then none of this would be an issue. You could happilyhave a stdlib::math and a mylib::math, and when you import either module your code will reflect that path explicitly. Hmm, this last sentence reminded me of something??? Oh yes -> "Explicit is better than implicit".


[1]: Yes, yes, i know about sys.path.insert. *puke*!

Chris Angelico 01-11-2013 06:30 AM

Re: PyWart: Import resolution order
 
On Fri, Jan 11, 2013 at 5:13 PM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> The fact that Python looks in the stdlib _first_ is not a good idea. It would seem more intuitive for a custom "math" module (living in the current directory) to /override/ the stlib "math" module. The proper order is as follows:
>
> 1. Current package or directory
> 2. stdlib
> 3. under the bed
> 4. who cares at this point


Why is it better to import from the current directory first? Windows
has that policy for executable commands; Unix, on the other hand,
requires that you put an explicit path for anything that isn't in the
standard search path. Which of these options is the more likely to
produce security holes and/or unexpected behaviour?

Welcome back to the list, Rick. Got any demonstrable code for Python 4000 yet?

ChrisA

Terry Reedy 01-11-2013 01:35 PM

Re: PyWart: Import resolution order
 
On 1/11/2013 1:13 AM, Rick Johnson wrote:
>
> Python's import resolution order is terrible.[1]
>
> The fact that Python looks in the stdlib _first_ is not a good idea.


And the fact is that it does not do so. The order depends on sys.path,
and '' is the first entry.

> It would seem more intuitive for a custom "math" module (living in
> the current directory) to /override/ the stlib "math" module.


Try it.

This is a nuisance though, when not intended. Someone writes a
random.py, and a year later in the same directory, an unrelated
hopscotch.py, which tries to import random.exponential. The import fails
and they post here, having forgotten about their own random.py, which
does not have such a function. Posts like this happen a few times a year.

--
Terry Jan Reedy


Michael Torrie 01-11-2013 05:53 PM

Re: PyWart: Import resolution order
 
On 01/10/2013 11:13 PM, Rick Johnson wrote:
>
> Python's import resolution order is terrible.[1]
>
> The fact that Python looks in the stdlib _first_ is not a good idea.


Whether or not the default behavior is desirable or not, sys.path is set
by default to look in the current directory first on any Python
distribution I have looked at. As Terry says, this fact causes a lot of
problems for newbies who accidentally override standard library modules
with their own python files and chaos ensues.

If your python installation does not search the current directory first,
then you must have changed sys.path.

Rick Johnson 01-12-2013 04:50 AM

Re: PyWart: Import resolution order
 
On Friday, January 11, 2013 7:35:37 AM UTC-6, Terry Reedy wrote:
> On 1/11/2013 1:13 AM, Rick Johnson wrote:
> > The fact that Python looks in the stdlib _first_ is not a good idea.

>
> And the fact is that it does not do so. The order depends on sys.path,
> and '' is the first entry.
>
> > It would seem more intuitive for a custom "math" module (living in
> > the current directory) to /override/ the stlib "math" module.

>
> This is a nuisance though, when not intended. Someone writes a
> random.py, and a year later in the same directory, an unrelated
> hopscotch.py, which tries to import random.exponential. The import fails
> and they post here, having forgotten about their own random.py, which
> does not have such a function. Posts like this happen a few times a year.


That's why i also mentioned the failure of Python to wrap stdlib modules in a package. If we would protect all built-in modules by placing them in a package (lib or py) then this problem would never happen.

Of course many people will **** and moan about the extra typing. I say, you have a choice: a few extra chars or multitudes of extra headaches -- I choose the first option.

Since more time is spent /maintaining/ code bases than /writing/ them, the explicit path is always the correct path to choose. Anyone who says otherwise is either careless or selfish (aka: seeking job security).

Rick Johnson 01-12-2013 05:28 AM

Re: PyWart: Import resolution order
 
On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
> Why is it better to import from the current directory first?


Opps. I was not explicit enough with my explanation :). I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


But the real solution is not to change the default resolution order. The real solution is to wrap all builtin modules into a package and use full paths to access every module. But wait, i have an even better idea... read below!

> Windows
> has that policy for executable commands; Unix, on the other hand,
> requires that you put an explicit path for anything that isn't in the
> standard search path. Which of these options is the more likely to
> produce security holes and/or unexpected behaviour?


I prefer the latter of course :).

I think if python where *strict* about full paths for non-builtins, then wewould be in a better place. But there is an even better solution! Force all python users to wrap THEIR modules in a toplevel package. Maybe even create the package for them. YES!. Call it "lib". Any "naked" modules (meaning modules that are not in a package) would have to be accessed starting from"lib". Of course professionals like you and i are already using packages to properly nest out modules, but the newbie's won't realize the power of packaging modules for some time, so create the default "lib" package for them..

For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".

So the conclusion is:

* We encourage python programmers to use packages so they avoid any name clashes with built-in modules.

* if they refuse fine, any "naked" modules they create will be packaged ina default package (call it "lib", "userlib", or whatever) and will requirethem to prefix the module name with "lib." -- or "lib:" if i get my way!

By doing this we solve the many problems related to module name resolution orders and we create cleaner code bases. Damn i am full of good ideas!

> Welcome back to the list, Rick. Got any demonstrable code
> for Python 4000 yet?


I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.

Rick Johnson 01-12-2013 05:28 AM

Re: PyWart: Import resolution order
 
On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
> Why is it better to import from the current directory first?


Opps. I was not explicit enough with my explanation :). I meant, "look in the current directory FIRST when in a package". Since many times (most all times) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


But the real solution is not to change the default resolution order. The real solution is to wrap all builtin modules into a package and use full paths to access every module. But wait, i have an even better idea... read below!

> Windows
> has that policy for executable commands; Unix, on the other hand,
> requires that you put an explicit path for anything that isn't in the
> standard search path. Which of these options is the more likely to
> produce security holes and/or unexpected behaviour?


I prefer the latter of course :).

I think if python where *strict* about full paths for non-builtins, then wewould be in a better place. But there is an even better solution! Force all python users to wrap THEIR modules in a toplevel package. Maybe even create the package for them. YES!. Call it "lib". Any "naked" modules (meaning modules that are not in a package) would have to be accessed starting from"lib". Of course professionals like you and i are already using packages to properly nest out modules, but the newbie's won't realize the power of packaging modules for some time, so create the default "lib" package for them..

For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".

So the conclusion is:

* We encourage python programmers to use packages so they avoid any name clashes with built-in modules.

* if they refuse fine, any "naked" modules they create will be packaged ina default package (call it "lib", "userlib", or whatever) and will requirethem to prefix the module name with "lib." -- or "lib:" if i get my way!

By doing this we solve the many problems related to module name resolution orders and we create cleaner code bases. Damn i am full of good ideas!

> Welcome back to the list, Rick. Got any demonstrable code
> for Python 4000 yet?


I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.

Chris Angelico 01-12-2013 06:03 AM

Re: PyWart: Import resolution order
 
On Sat, Jan 12, 2013 at 4:28 PM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
>> Welcome back to the list, Rick. Got any demonstrable code
>> for Python 4000 yet?

>
> I am working on it. Stay tuned. Rick is going to rock your little programming world /very/ soon.


So all I have to do is paper my programming world and I win?

Sorry, can't hang around arguing about module search paths and your
recommendations to add piles of syntactic salt. Gotta finish building
and playing with today's Linux kernel (3.2.7, it's looking good so
far).

ChrisA

Ian Kelly 01-12-2013 07:36 AM

Re: PyWart: Import resolution order
 
On Fri, Jan 11, 2013 at 10:28 PM, Rick Johnson
<rantingrickjohnson@gmail.com> wrote:
> On Friday, January 11, 2013 12:30:27 AM UTC-6, Chris Angelico wrote:
>> Why is it better to import from the current directory first?

>
> Opps. I was not explicit enough with my explanation :). I meant, "look inthe current directory FIRST when in a package". Since many times (most alltimes) packages will contain many sub-modules that need to be imported into the package's main.py module, and sometimes these modules will have the same name as a stdlib module, then looking in the package FIRST makes sense.


And again, in Python 2.x this is already the case. When importing in
a package, it tries to do a relative import before it even looks at
sys.path.

> I think if python where *strict* about full paths for non-builtins, then we would be in a better place.


And again, in Python 3, where implicit relative imports have been
removed from the language, it already is strict about using full
paths. You can still do relative imports, but you have to be explicit
about them.

> For instance you could create a package named "chris" and then have a module named math exist inside. Alternatively if you choose to be a non-professional and create a math module without a containing package, python would throw the module into the default "lib" package. The only way you could access your math module now would be by using the path "lib.math".


What if I create a package named "math"? Does that also automatically
get renamed to "lib.math"? How is it decided what package names are
proper; is it just because it happens to clash with a stdlib name that
the package gets magically renamed?

What if I create a package, and then later a module with the same name
happens to be added to the stdlib? My program that uses the package
just breaks because it no longer imports the correct thing?

> Damn i am full of good ideas!


Your ideas might be better if you first spent some time gaining a
better understanding of how the language works as is.

alex23 01-12-2013 07:44 AM

Re: PyWart: Import resolution order
 
On 12 Jan, 14:50, Rick Johnson <rantingrickjohn...@gmail.com> wrote:
> Of course many people will **** and moan about the extra typing.


You just ignored the fact that your original claim was incorrect and
kept going on with your rant anyway.

> Since more time is spent /maintaining/ code bases than /writing/ them


In your case, more time is actually spent on insisting _other_ people
maintain code bases.

Everyone: PLEASE STOP FEEDING THE TROLL


All times are GMT. The time now is 05:51 AM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.