Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Import from database

Reply
Thread Tools

Import from database

 
 
Steve Holden
Guest
Posts: n/a
 
      01-24-2005
I'm trying to load module code from a database, which stores for each
module its full name, code, load date and a Boolean indicating whether
it's a package or not.

The following simple program:

import dbimp, sys

if __name__ == "__main__":
dbimp.install()
#import bsddb.db
import a.b.c.d
import bsddb

gives a traceback from its last line. The whole output is

$ python -i test.py
Accepted *db*
found a in db
load_module: a
a loaded: <module 'a' from 'db:a'> pkg: 1
found a.b in db
load_module: a.b
a.b loaded: <module 'a.b' from 'db:a.b'> pkg: 1
found a.b.c in db
load_module: a.b.c
a.b.c loaded: <module 'a.b.c' from 'db:a.b.c'> pkg: 1
found a.b.c.d in db
load_module: a.b.c.d
a.b.c.d loaded: <module 'a.b.c.d' from 'db:a.b.c.d'> pkg: 0
found bsddb in db
load_module: bsddb
found weakref in db
load_module: weakref
weakref loaded: <module 'weakref' from 'db:weakref'> pkg: 0
Traceback (most recent call last):
File "test.py", line 7, in ?
import bsddb
File "/c/steve/Projects/Python/dbimp/dbimp.py", line 49, in load_module
exec code in module.__dict__
File "db:bsddb", line 62, in ?
File "/usr/lib/python2.4/os.py", line 133, in ?
from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
altsep,
ImportError: No module named path
>>>


It appears that for some reason execution of the code in
bsddb/__init__.pyc (loaded from the database using my importer) is
causing the os module to execute again although it has already run and
been cached in sys.modules.

Any explanations would be welcome. The importer module is shown below (a
posting with an attachment to avoid code folding didn't appear to make
it out)

Also if someone can suggest how associate the source file with
module/package code from the database that would make tracebacks look
more conventional, though I'd like to retain the indication the module
was loaded by dbimp somehow.

#
# Import modules from a database
#
import sys, db, marshal, imp, new
conn = db.conn()
curs = conn.cursor()

curs.execute("select modName from module")
impdict = {}
for n in [x[0] for x in curs.fetchall()]:
impdict[n] = 1


class dbimporter(object):

def __init__(self, item, *args, **kw):
if item != "*db*":
raise ImportError
print "Accepted", item

def find_module(self, fullname, path=None):
#print "find_module:", fullname, "from", path
if fullname not in impdict:
#print "Bailed on", fullname
return None
else:
print "found", fullname, "in db"
return self

def load_module(self, modname):
print "load_module:", modname
if modname in sys.modules:
return sys.modules[modname]
curs.execute("select modCode, modPackage from module where
modName=%s", (modname, ))
row = curs.fetchone() # should only BE one ...S
if not row:
#print modname, "not found in db"
raise ImportError, "DB module %s not found in modules"
code, package = row
code = marshal.loads(code)
module = new.module(modname)
sys.modules[modname] = module
module.__name__ = modname
exec code in module.__dict__
module.__file__ = "db:%s" % modname
module.__loader__ = dbimporter
if package:
module.__path__ = sys.path
exec code in module.__dict__
print modname, "loaded:", repr(module), "pkg:", package
return module

def install():
sys.path_hooks.append(dbimporter)
sys.path_importer_cache.clear() # probably not necessary
sys.path.insert(0, "*db*") # probably not needed with a metea-path
hook?


regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
 
Reply With Quote
 
 
 
 
Kartic
Guest
Posts: n/a
 
      01-24-2005
Steve,

I believe you have to put ntpath, macpath and posixpath in the module
database for os.path to work.

I tried it with zipimporter builtin and I got the same traceback till I
added ntpath.py to my zip file. (Of course, I renamed the original
ntpath to _ntpath so that the original did not get imported)
Thanks,
--Kartic

 
Reply With Quote
 
 
 
 
Steve Holden
Guest
Posts: n/a
 
      01-25-2005
Kartic wrote:

> Steve,
>
> I believe you have to put ntpath, macpath and posixpath in the module
> database for os.path to work.
>
> I tried it with zipimporter builtin and I got the same traceback till I
> added ntpath.py to my zip file. (Of course, I renamed the original
> ntpath to _ntpath so that the original did not get imported)
> Thanks,
> --Kartic
>

I'm not sure I understand, as I am currently relying on the system
implementations of any modules that happen to be loaded before my code
picks up. As to what's in the database

SELECT *
FROM `module`
WHERE modName LIKE '%path%'

shows that I have macpathm macurlpath, ntpath, nturlpath, os2emxpath and
posixpath all there (but, as I say, I've already executed the standard
modules by the time anything of mine gets to run). If I alter my test
program to:

import dbimp, sys

if __name__ == "__main__":
dbimp.install()
k = sys.modules.keys()
k.sort()
for kk in k:
print kk
#import bsddb.db
import a.b.c.d
import bsddb

then I get as output

MySQLdb
MySQLdb.MySQLdb
...
MySQLdb.types
UserDict
__builtin__
__main__
_codecs
_locale
_mysql
_mysql_exceptions
_sre
array
cPickle
codecs
copy
copy_reg
db
dbimp
...
mx.Misc
mx.Misc.LazyModule
new
nt
ntpath
os
os.path
re
...
zipimport
Accepted *db*
found a in db
load_module: a
a loaded: <module 'a' from 'db:a'> pkg: 1
found a.b in db
load_module: a.b
a.b loaded: <module 'a.b' from 'db:a.b'> pkg: 1
found a.b.c in db
load_module: a.b.c
a.b.c loaded: <module 'a.b.c' from 'db:a.b.c'> pkg: 1
found a.b.c.d in db
load_module: a.b.c.d
a.b.c.d loaded: <module 'a.b.c.d' from 'db:a.b.c.d'> pkg: 0
found bsddb in db
load_module: bsddb
found weakref in db
load_module: weakref
weakref loaded: <module 'weakref' from 'db:weakref'> pkg: 0
Traceback (most recent call last):
File "test.py", line 11, in ?
import bsddb
File "/c/steve/Projects/Python/dbimp/dbimp.py", line 49, in load_module
exec code in module.__dict__
File "db:bsddb", line 62, in ?
File "/usr/lib/python2.4/os.py", line 133, in ?
from os.path import (curdir, pardir, sep, pathsep, defpath, extsep,
altsep,
ImportError: No module named path

In other words, the os module is /already/ in sys.smodules, as is
os.path, yet the interpreter is complaining (I presume) that os.path is
not a module. I don't even know *why* os is being executed a second
time. I can only assume it's being imported as some other name like
"bsddb.os" and some element of the import system is actually doing an
import rather than refusing to guess.

I presume I need to control the import process more closely to make sure
that this import attempt is rejected, but I can't see how to do that.

regards
Steve

 
Reply With Quote
 
Kartic
Guest
Posts: n/a
 
      01-25-2005
Steve,

Hmmm... Yes, I see what you are saying. Could you post your create
table statement so that I can create a db and play around with
dbimport?

Thanks,
--Kartic

 
Reply With Quote
 
Steve Holden
Guest
Posts: n/a
 
      01-25-2005
Kartic wrote:

> Steve,
>
> Hmmm... Yes, I see what you are saying. Could you post your create
> table statement so that I can create a db and play around with
> dbimport?
>
> Thanks,
> --Kartic
>

Here it is, plus also the loader program I used to suck in the standard
library (in case it's that that's faulty):

CREATE TABLE `module` (
`modName` varchar(25) NOT NULL default '',
`modPackage` tinyint(1) NOT NULL default '0',
`modCompileTime` timestamp(14) NOT NULL,
`modCode` longtext,
PRIMARY KEY (`modName`)
) TYPE=MyISAM;

#### WARNING: email client may wrap some lines ...
#
# Establish standard library in database
#
import db
import os
import glob
import sys
import marshal

conn = db.conn()
curs = conn.cursor()

if sys.argv[1] == "-d":
curs.execute("delete from module")
print "All existing modules cleared"
del sys.argv[1]

def importpy(path, modname, package):
print "Importing", path, modname
c = compile(file(path).read(), "db:%s" % modname, "exec")
curs.execute("""delete from module where modName = %s""", (modname,))
curs.execute("""insert into module (modName, modCode, modPackage,
modCompileTime)
values (%s, %s, %s, now())""", (modname,
marshal.dumps(c), package))
print "Added", modname
conn.commit()

def importall(path, modlist):
os.chdir(path)
for f in glob.glob("*"):
if os.path.isdir(f):
fn = os.path.join(path, f, "__init__.py")
if os.path.exists(fn):
ml = modlist + [f]
importpy(fn, ".".join(ml), 1)
importall(os.path.join(path, f), ml)
elif f.endswith('.py') and '.' not in f[:-3] and f !=
"__init__.py":
importpy(os.path.join(path, f), ".".join(modlist+[f[:-3]]), 0)

if __name__ == "__main__":
for path in sys.argv[1:]:
importall(path, [])

regards
Steve

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Database Database Database Database scott93727@gmail.com Computer Information 0 09-27-2012 02:43 AM
DataBase DataBase DataBase DataBase scott93727@gmail.com Computer Information 0 09-26-2012 09:40 AM
to use import java.lang.* or import java.lang.Math or none at all? JPractitioner Java 13 02-24-2006 08:48 PM
XML Schema question - does "import" import elements? Vitali Gontsharuk XML 2 08-25-2005 07:33 PM
IMPORT STATIC; Why is "import static" file scope? Why not class scope? Paul Opal Java 12 10-10-2004 11:01 PM



Advertisments