Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > module import search path strangeness

Reply
Thread Tools

module import search path strangeness

 
 
tow
Guest
Posts: n/a
 
      08-11-2008
I have a python script (part of a django application, if it makes any
difference) which is exhibiting the following behaviour:

import my_module # succeeds
imp.find_module("my_module") # fails, raising ImportError

which is completely baffling me. According to sys.path, both should
fail; the directory containing my_module is not in sys.path (though
the my_module directory itself is). More puzzlingly, printing out
my_module.__file__ gives:

/home/tow/test/my_module/../my_module/__init__.pyc

I don't really understand what the ".." is doing in there.

Can someone explain what I'm missing here, it's got me stumped.

Toby
 
Reply With Quote
 
 
 
 
Gabriel Genellina
Guest
Posts: n/a
 
      08-12-2008
En Mon, 11 Aug 2008 19:19:19 -0300, tow <(E-Mail Removed)>
escribi�:

> I have a python script (part of a django application, if it makes any
> difference) which is exhibiting the following behaviour:
>
> import my_module # succeeds
> imp.find_module("my_module") # fails, raising ImportError
>
> which is completely baffling me. According to sys.path, both should
> fail; the directory containing my_module is not in sys.path (though
> the my_module directory itself is).


my_module is not a module but a package, right? Else I don't understand
the above statement.

> More puzzlingly, printing out
> my_module.__file__ gives:
>
> /home/tow/test/my_module/../my_module/__init__.pyc
>
> I don't really understand what the ".." is doing in there.
>
> Can someone explain what I'm missing here, it's got me stumped.


Perhaps you have ".." in sys.path? And the current directory happens to be
/home/tow/test/my_module?

--
Gabriel Genellina

 
Reply With Quote
 
 
 
 
tow
Guest
Posts: n/a
 
      08-12-2008
On Aug 12, 4:59*am, "Gabriel Genellina" <(E-Mail Removed)>
wrote:
> En Mon, 11 Aug 2008 19:19:19 -0300, tow <(E-Mail Removed)> *
> escribi :
>
> > I have a python script (part of a django application, if it makes any
> > difference) which is exhibiting the following behaviour:

>
> > import my_module # succeeds
> > imp.find_module("my_module") # fails, raising ImportError

>
> > which is completely baffling me. According to sys.path, both should
> > fail; the directory containing my_module is not in sys.path (though
> > the my_module directory itself is).

>
> my_module is not a module but a package, right? Else I don't understand *
> the above statement.


Sorry, sloppy terminology on my part, yes, my_module is a package.

> > More puzzlingly, printing out
> > my_module.__file__ gives:

>
> > /home/tow/test/my_module/../my_module/__init__.pyc

>
> > I don't really understand what the ".." is doing in there.

>
> > Can someone explain what I'm missing here, it's got me stumped.

>
> Perhaps you have ".." in sys.path? And the current directory happens to be *
> /home/tow/test/my_module?


The current directory is actually a subdirectory of /home/tow/test/
my_module,
but ".." is not in sys.path (nor is anything containing "..") In any
case, if
it were a matter of sys.path, surely the imp.find_module call above
should succeed?

Basically, I had thought that import and imp.find_module used exactly
the same
search path, but the above example shows that at least in this
circumstance they
don't; import is picking up additional search paths from somewhere -
what am I missing?

Toby
 
Reply With Quote
 
Peter Otten
Guest
Posts: n/a
 
      08-12-2008
tow wrote:

>>> > I have a python script (part of a django application, if it makes any
>> > difference) which is exhibiting the following behaviour:

>>
>> > import my_module # succeeds
>> > imp.find_module("my_module") # fails, raising ImportError

>>
>> > which is completely baffling me. According to sys.path, both should
>> > fail; the directory containing my_module is not in sys.path (though
>> > the my_module directory itself is).


> The current directory is actually a subdirectory of /home/tow/test/
> my_module,
> but ".." is not in sys.path (nor is anything containing "..") In any


> Basically, I had thought that import and imp.find_module used exactly
> the same
> search path, but the above example shows that at least in this
> circumstance they
> don't; import is picking up additional search paths from somewhere -
> what am I missing?


Grepping through the django source finds

../trunk/django/core/management/__init__.py:
sys.path.append(os.path.join(project_directory, os.pardir))

sys.path could be changed as a side effect of an import, so I ask you to
verify (again) that import and imp.find_module() see the same sys.path.

assert all(".." not in p for p in sys.path)
import my_module

assert all(".." not in p for p in sys.path)
imp.find_module("my_module")


Peter
 
Reply With Quote
 
tow
Guest
Posts: n/a
 
      08-12-2008
On Aug 12, 9:56*am, Peter Otten <(E-Mail Removed)> wrote:
> tow wrote:


> > Basically, I had thought that import and imp.find_module used exactly
> > the same
> > search path, but the above example shows that at least in this
> > circumstance they
> > don't; import is picking up additional search paths from somewhere -
> > what am I missing?

>
> Grepping through the django source finds
>
> ./trunk/django/core/management/__init__.py: *
> sys.path.append(os.path.join(project_directory, os.pardir))


Hmm. It turns out that that is indeed the issue, but in a way that
wasn't immediately obvious to me. Looking at it in more context:

sys.path.append(os.path.join(project_directory, os.pardir))
project_module = __import__(project_name, {}, {}, [''])
sys.path.pop()

sys.path is extended, the project module is imported, then the
additional path is dropped from sys.path.
And if I comment out those sys.path manipulations, I get the result I
expect later on.

What I think is happening is that in that stanza, project_module
(which is my_module in my case) is
imported from the altered sys.path. sys.path is then put back, but
python now knows about my_module.

As a result when my_module is imported again elsewhere, python already
knows about it, so no searching
of sys.path is done (which is good, because it wouldn't be found on
the current sys.path), and the
import apparently succeeds, though in fact it's effectively
(actually?) a no-op.

However, imp.find_module("my_module") forces a search of the sys.path,
and thus fails.

Or at least, that is what I surmise, given that I see

import my_module # succeeds
imp.find_module("my_module") # fails, raising ImportError

So, to answer my original question, the difference in search behaviour
between "import" and "imp.find_module" is that the former might not
look at sys.path at all if the module has already been loaded, while
the latter will only search on the current sys.path.

Am I right?

Toby
 
Reply With Quote
 
Peter Otten
Guest
Posts: n/a
 
      08-12-2008
tow wrote:

> On Aug 12, 9:56Â*am, Peter Otten <(E-Mail Removed)> wrote:
>> tow wrote:

>
>> > Basically, I had thought that import and imp.find_module used exactly
>> > the same
>> > search path, but the above example shows that at least in this
>> > circumstance they
>> > don't; import is picking up additional search paths from somewhere -
>> > what am I missing?

>>
>> Grepping through the django source finds
>>
>> ./trunk/django/core/management/__init__.py:
>> sys.path.append(os.path.join(project_directory, os.pardir))

>
> Hmm. It turns out that that is indeed the issue, but in a way that
> wasn't immediately obvious to me. Looking at it in more context:
>
> sys.path.append(os.path.join(project_directory, os.pardir))
> project_module = __import__(project_name, {}, {}, [''])
> sys.path.pop()


Ouch.

> So, to answer my original question, the difference in search behaviour
> between "import" and "imp.find_module" is that the former might not
> look at sys.path at all if the module has already been loaded, while
> the latter will only search on the current sys.path.
>
> Am I right?


Yes. 'import' looks up the file in a cache, the sys.modules dictionary,
before it falls back to the more costly alternatives.

Peter
 
Reply With Quote
 
Sion Arrowsmith
Guest
Posts: n/a
 
      08-12-2008
Peter Otten <(E-Mail Removed)> wrote:
>tow wrote:
>> sys.path.append(os.path.join(project_directory, os.pardir))
>> project_module = __import__(project_name, {}, {}, [''])
>> sys.path.pop()

>Ouch.


I presume that "Ouch" is in consideration of what might happen if
the subject of the __import__ also calls sys.path.append ...?

--
\S -- http://www.velocityreviews.com/forums/(E-Mail Removed) -- http://www.chaos.org.uk/~sion/
"Frankly I have no feelings towards penguins one way or the other"
-- Arthur C. Clarke
her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump
 
Reply With Quote
 
tow
Guest
Posts: n/a
 
      08-12-2008
On Aug 12, 4:59*pm, Peter Otten <(E-Mail Removed)> wrote:
> tow wrote:
> > On Aug 12, 9:56*am, Peter Otten <(E-Mail Removed)> wrote:
> >> tow wrote:

>
> >> > Basically, I had thought that import and imp.find_module used exactly
> >> > the same
> >> > search path, but the above example shows that at least in this
> >> > circumstance they
> >> > don't; import is picking up additional search paths from somewhere -
> >> > what am I missing?

>
> >> Grepping through the django source finds

>
> >> ./trunk/django/core/management/__init__.py:
> >> sys.path.append(os.path.join(project_directory, os.pardir))

>
> > Hmm. It turns out that that is indeed the issue, but in a way that
> > wasn't immediately obvious to me. Looking at it in more context:

>
> > * * sys.path.append(os.path.join(project_directory, os.pardir))
> > * * project_module = __import__(project_name, {}, {}, [''])
> > * * sys.path.pop()

>
> Ouch.
>
> > So, to answer my original question, the difference in search behaviour
> > between "import" and "imp.find_module" is that the former might not
> > look at sys.path at all if the module has already been loaded, while
> > the latter will only search on the current sys.path.

>
> > Am I right?

>
> Yes. 'import' looks up the file in a cache, the sys.modules dictionary,
> before it falls back to the more costly alternatives.
>
> Peter


Thanks, at least I understand what's going on now.

Toby
 
Reply With Quote
 
Peter Otten
Guest
Posts: n/a
 
      08-12-2008
Sion Arrowsmith wrote:

> Peter Otten <(E-Mail Removed)> wrote:
>>tow wrote:
>>> sys.path.append(os.path.join(project_directory, os.pardir))
>>> project_module = __import__(project_name, {}, {}, [''])
>>> sys.path.pop()

>>Ouch.

>
> I presume that "Ouch" is in consideration of what might happen if
> the subject of the __import__ also calls sys.path.append ...?


Yes

Peter
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
imp.find_module don't found my module but standard import statement can import this module… why ? Stéphane Klein Python 0 07-05-2011 07:36 AM
Why 'import module' will not import module.py but the directorymodule? Peng Yu Python 0 10-31-2009 11:51 PM
csv module strangeness. tobiah Python 16 08-31-2006 09:44 PM
Handling import conflicts when module has the same name as a library module that it needs to import? plb Python 0 02-08-2005 01:08 PM
Handling import conflicts when module has the same name as a library module that it needs to import? plb Python 0 02-08-2005 01:01 PM



Advertisments