Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Loading multiple versions of the same package at the same time

Reply
Thread Tools

Loading multiple versions of the same package at the same time

 
 
della
Guest
Posts: n/a
 
      11-27-2008
Hi all,

I've got some pickled files representing graphs (using networkx,
http://networkx.lanl.gov if you're interested) that were produced
using version 0.36 of the library.

Now, they have released a new version of the library which is
incompatible with respect to pickled files, so what I'd like to do is
to write a script that loads the two versions of the library at once,
unpickles with the old one, creates a new object with the new version
and re-pickles it with the new version.

So, I installed the two versions of the package using easy_install -m,
but it looks like I can't import the two versions at once:

>>> from pkg_resources import require
>>> require('networkx==0.36')

[networkx 0.36 (/usr/lib/python2.5/site-packages/networkx-0.36-
py2.5.egg)]
>>> import networkx as nx_old
>>> require('networkx==1.0')

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626,
in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 528,
in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (networkx 0.36 (/usr/lib/python2.5/site-
packages/networkx-0.36-py2.5.egg), Requirement.parse('networkx==1.0'))

Any ideas?
Thanks a lot
matteo
 
Reply With Quote
 
 
 
 
Diez B. Roggisch
Guest
Posts: n/a
 
      11-27-2008
della wrote:

> Hi all,
>
> I've got some pickled files representing graphs (using networkx,
> http://networkx.lanl.gov if you're interested) that were produced
> using version 0.36 of the library.
>
> Now, they have released a new version of the library which is
> incompatible with respect to pickled files, so what I'd like to do is
> to write a script that loads the two versions of the library at once,
> unpickles with the old one, creates a new object with the new version
> and re-pickles it with the new version.
>
> So, I installed the two versions of the package using easy_install -m,
> but it looks like I can't import the two versions at once:
>
>>>> from pkg_resources import require
>>>> require('networkx==0.36')

> [networkx 0.36 (/usr/lib/python2.5/site-packages/networkx-0.36-
> py2.5.egg)]
>>>> import networkx as nx_old
>>>> require('networkx==1.0')

> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 626,
> in require
> needed = self.resolve(parse_requirements(requirements))
> File "/usr/lib/python2.5/site-packages/pkg_resources.py", line 528,
> in resolve
> raise VersionConflict(dist,req) # XXX put more info here
> pkg_resources.VersionConflict: (networkx 0.36 (/usr/lib/python2.5/site-
> packages/networkx-0.36-py2.5.egg), Requirement.parse('networkx==1.0'))
>
> Any ideas?


You can't do that. How should python distinguish between the two modules
with the same name?

What you can do is

- import the old package
- unpickle
- convert to some neutral exchange format, simple dicts, lists, tuples
- pickle that

- import the new package
- unpickle the exchange data
- stuff it into the new classes

Depending on who's got control for what, you might consider overloading the
pickle protocol to allow for this more automatic in the future.

This whole ruckus is BTW one of the reasons I moved away from ZODB.

Diez
 
Reply With Quote
 
 
 
 
della
Guest
Posts: n/a
 
      11-27-2008
On 27 Nov, 11:21, "Diez B. Roggisch" <(E-Mail Removed)> wrote:

> You can't do that. How should python distinguish between the two modules
> with the same name?


That's why I was trying to import them with different names

> What you can do is
>
> *- import the old package
> *- unpickle
> *- convert to some neutral exchange format, simple dicts, lists, tuples
> *- pickle that
>
> *- import the new package
> *- unpickle the exchange data
> *- stuff it into the new classes


Thanks, this is what I did.

matteo
 
Reply With Quote
 
Diez B. Roggisch
Guest
Posts: n/a
 
      11-27-2008
della wrote:

> On 27 Nov, 11:21, "Diez B. Roggisch" <(E-Mail Removed)> wrote:
>
>> You can't do that. How should python distinguish between the two modules
>> with the same name?

>
> That's why I was trying to import them with different names


You weren't. The "as" creates just a local alias. It's roughly equivalent to


import foo as bar

<->

import foo
bar = foo
del foo


But in the interpreters module dict, foo it is, and stays.

Diez
 
Reply With Quote
 
Terry Reedy
Guest
Posts: n/a
 
      11-27-2008
Diez B. Roggisch wrote:
> della wrote:
>
>> On 27 Nov, 11:21, "Diez B. Roggisch" <(E-Mail Removed)> wrote:
>>
>>> You can't do that. How should python distinguish between the two modules
>>> with the same name?

>> That's why I was trying to import them with different names

>
> You weren't. The "as" creates just a local alias. It's roughly equivalent to
>
>
> import foo as bar
>
> <->
>
> import foo
> bar = foo
> del foo
>
>
> But in the interpreters module dict, foo it is, and stays.


But giving the modules different names on the disk should work, no?

 
Reply With Quote
 
della
Guest
Posts: n/a
 
      11-28-2008
On 27 Nov, 21:34, Terry Reedy <(E-Mail Removed)> wrote:

> > But in the interpreters module dict, foo it is, and stays.

>
> But giving the modules different names on the disk should work, no?


Yes, but -- for what I've understood -- that wouldn't solve my
original problem with pickle, since I would need to carry around a
module with the new name forever

matteo
 
Reply With Quote
 
Peter Otten
Guest
Posts: n/a
 
      11-28-2008
della wrote:

> On 27 Nov, 21:34, Terry Reedy <(E-Mail Removed)> wrote:
>
>> > But in the interpreters module dict, foo it is, and stays.

>>
>> But giving the modules different names on the disk should work, no?

>
> Yes, but -- for what I've understood -- that wouldn't solve my
> original problem with pickle, since I would need to carry around a
> module with the new name forever


You can rename the globals within the pickle. A starting point, not tested
beyond running the demo script:

$ cat fixpickle.py
import pickle
import pickletools

def ops(d):
prevpos = None
previnfo = None
for info, arg, pos in pickletools.genops(d):
if prevpos is not None:
yield d[prevposos]
prevpos = pos
previnfo = info
yield d[prevpos:]


def tocode(dotted):
parts = tuple(dotted.rsplit(".", 1))
return "c%s\n%s\n" % parts

def rename_globals(d, pairs):
updates = dict((tocode(old), tocode(new)) for old, new in pairs)
return "".join(updates.get(o, o) for o in ops(d))

$ cat alpha.py
class A(object):
def __init__(self, x, y):
self.x = x
self.y = y
def __str__(self):
return "A(%s, %s)" % (self.x, self.y)

$ cat beta.py
class B(object):
def __str__(self):
return "B(%s, %s)" % (self.x, self.y)

$ cat demo.py
import alpha
import fixpickle
import pickle

if __name__ == "__main__":
a = alpha.A(1, 2)
print a
d = pickle.dumps(a)
d = fixpickle.rename_globals(d, [("alpha.A", "beta.B")])
b = pickle.loads(d)
print b

Peter
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Multiple versions of Python coexisting in the same OS Edward Diener Python 36 07-26-2010 05:37 PM
Running multiple versions of Python on the same host.. Cowmix Python 3 07-10-2006 06:04 PM
Multiple versions of Python / PythonWin on the same machine? dananrg@yahoo.com Python 2 02-24-2006 11:17 PM
Loading Two Different Versions of the Same Class/JAR??? John Davison Java 3 07-16-2004 08:02 AM
How to use 2 DIFFERENT VERSIONS of the SAME MODULE in the same perl program ? Yves Petinot Perl Misc 9 06-30-2004 08:26 AM



Advertisments