Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Import a module from a non-file?

Reply
Thread Tools

Import a module from a non-file?

 
 
Petri Savolainen
Guest
Posts: n/a
 
      10-17-2003
I was trying to roll my own python code importer, but in the end, it seems
that no matter what you try, it is always necessary to supply a REAL file
object for python lower-level import machinery to work. Is this really
true? Or did I miss something?

Of course, it is easy to get the module source from somewhere, put it into a
tmpfile etc. ... but I'd be nice to be able to skip that extra step.

Thanks,

Petri
 
Reply With Quote
 
 
 
 
Peter Hansen
Guest
Posts: n/a
 
      10-17-2003
Petri Savolainen wrote:
>
> I was trying to roll my own python code importer, but in the end, it seems
> that no matter what you try, it is always necessary to supply a REAL file
> object for python lower-level import machinery to work. Is this really
> true? Or did I miss something?
>
> Of course, it is easy to get the module source from somewhere, put it into a
> tmpfile etc. ... but I'd be nice to be able to skip that extra step.


You don't say what you actually tried. Doesn't imp.load_module() allow
passing a "file-like object" containing the source?

What about exec? I would think that if you read in the source code string,
created a new module, and passed its dictionary in to exec, you would get
basically the same result.

(Neither idea is based on personal success doing this... they're just ideas.)

-Peter
 
Reply With Quote
 
 
 
 
Alex Martelli
Guest
Posts: n/a
 
      10-17-2003
Petri Savolainen wrote:

> I was trying to roll my own python code importer, but in the end, it seems
> that no matter what you try, it is always necessary to supply a REAL file
> object for python lower-level import machinery to work. Is this really
> true? Or did I miss something?


No, it's not true. Read PEP 302, http://www.python.org/peps/pep-0302.html ,
for details. Basically, you can write and install "Importer" objects,
which may return "Loader" objects (an importer an also be a loader, if
you wish, so importer.find_module can do a "return self" as long as the
importer object also has a load_module method). A loader's method
load_module must return the fully loaded module object, and it can
built it in any way it pleases -- e.g. with new.module from standard
module new to make a new empty module object, then (if for example it
has obtained the Python source of the module from wherever) an exec
of that source using the new module's dictionary as locals and globals.

Let me give a trivial example made by overriding __import__ -- NOT the
recommended mechanism, just simpler to explain!

import __builtin__
bi = __builtin__.__import__

def __import__(name, *args):
if name == 'foo': return makefoo()
return bi(name, *args)
__builtin__.__import__ = __import__

import new
import sys
def makefoo():
mod = new.module('foo')
source = '''
print "foo is being imported"
def foo(): return "hi, foo here"
'''
exec source in mod.__dict__
return mod

import foo
print foo.foo()
import foo
print foo.foo()

running this emits:

[alex@lancelot ba]$ python i.py
foo is being imported
hi, foo here
foo is being imported
hi, foo here
[alex@lancelot ba]$

note that each import invokes makefoo AGAIN: if you want to use
the cache conveniently maintained for you in sys.modules['foo']
after the first time foo is imported, you have to do that
explicitly should you choose to use this "override __import__"
old-but-still-there functionality (caching for the new and
improved mechanisms of PEP 302 is different -- again, see the
PEP itself for all details).


Alex

 
Reply With Quote
 
David Boddie
Guest
Posts: n/a
 
      10-17-2003
Alex Martelli <(E-Mail Removed)> wrote in message news:<sAUjb.7504$(E-Mail Removed)>...
> Petri Savolainen wrote:
>
> > I was trying to roll my own python code importer, but in the end, it seems
> > that no matter what you try, it is always necessary to supply a REAL file
> > object for python lower-level import machinery to work. Is this really
> > true? Or did I miss something?

>
> No, it's not true. Read PEP 302, http://www.python.org/peps/pep-0302.html ,
> for details.


That's only for Python 2.3 and later, though, isn't it? I ran into this problem
using Python 2.2 with the ihooks and imp modules. I don't remember exactly
which part of the infrastructure was insisting on file objects but the following
lines from Python/import.c in the Python 2.3 source distribution would appear
to be relevant:

/* First check that there's an open file (if we need one) */
switch (type) {
case PY_SOURCE:
case PY_COMPILED:
if (fp == NULL) {
PyErr_Format(PyExc_ValueError,
"file object required for import (type code %d)",
type);
return NULL;
}
}

I really should investigate the new import mechanism. In the meantime, it's good
to know that the alternative import hook solutions still work with Python 2.3.

David
 
Reply With Quote
 
Alex Martelli
Guest
Posts: n/a
 
      10-18-2003
David Boddie wrote:

> Alex Martelli <(E-Mail Removed)> wrote in message
> news:<sAUjb.7504$(E-Mail Removed)>...
>> Petri Savolainen wrote:
>>
>> > I was trying to roll my own python code importer, but in the end, it
>> > seems that no matter what you try, it is always necessary to supply a
>> > REAL file object for python lower-level import machinery to work. Is
>> > this really true? Or did I miss something?

>>
>> No, it's not true. Read PEP 302,
>> http://www.python.org/peps/pep-0302.html , for details.

>
> That's only for Python 2.3 and later, though, isn't it? I ran into this


Yes, PEP 302 and its early benefits (import-from-zip) were first implemented
with Python 2.3.

> problem using Python 2.2 with the ihooks and imp modules. I don't remember


Admittedly not the cleanest architecture, though quite workable.

> exactly which part of the infrastructure was insisting on file objects but
> the following lines from Python/import.c in the Python 2.3 source
> distribution would appear to be relevant:
>
> /* First check that there's an open file (if we need one) */
> switch (type) {
> case PY_SOURCE:
> case PY_COMPILED:
> if (fp == NULL) {
> PyErr_Format(PyExc_ValueError,
> "file object required for import (type code %d)",
> type);
> return NULL;
> }
> }


That's the load_module function exposed by module imp -- and in
any case when your type is IMP_HOOK you don't enter this test (in
2.3 ff -- no older sources around to check, sorry).


> I really should investigate the new import mechanism. In the meantime,
> it's good to know that the alternative import hook solutions still work
> with Python 2.3.


You mean overriding the builtin __import__ ? Yes, it's kept around
for backwards compatibility, of course (also weird cases in which you
WANT the sys.modules cache to be systematically bypassed, though I
admit I can't think of a use case for that right now....


Alex

 
Reply With Quote
 
Irmen de Jong
Guest
Posts: n/a
 
      10-18-2003
Petri Savolainen wrote:

> I was trying to roll my own python code importer, but in the end, it seems
> that no matter what you try, it is always necessary to supply a REAL file
> object for python lower-level import machinery to work. Is this really
> true? Or did I miss something?


Probably, because I have been importing modules from over a network
socket for a long time now in Pyro. (http://pyro.sourceforge.net)

The code in Pyro works back to python version 2.0, if I'm not mistaken,
so you won't need to upgrade to 2.3 and use all new kinds of import hooks...

Basically what I'm doing is
* load the module's bytecode over the network
* create a new module and put it in sys.modules using new.module(...)
* execute the downloaded code in this new module: exec code in mod.__dict__
* some extra magic to trap cascaded imports.

If you need more info, have a look at the _retrieveCode method
in the protocol.py file of Pyro, or just ask

I'm not sure if my way of doing things is the 'best' way, but hey,
it works...

--Irmen de Jong

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
imp.find_module don't found my module but standard import statement can import this module… why ? Stéphane Klein Python 0 07-05-2011 07:36 AM
Why 'import module' will not import module.py but the directorymodule? Peng Yu Python 0 10-31-2009 11:51 PM
Handling import conflicts when module has the same name as a library module that it needs to import? plb Python 0 02-08-2005 01:08 PM
Handling import conflicts when module has the same name as a library module that it needs to import? plb Python 0 02-08-2005 01:01 PM
import vs from module import : any performance issue? Pierre Rouleau Python 4 03-07-2004 05:06 PM



Advertisments