Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re-executing the code object from a class 'declaration'

Reply
Thread Tools

Re-executing the code object from a class 'declaration'

 
 
Carlos Ribeiro
Guest
Posts: n/a
 
      10-06-2004
As part of a pet project of mine, I needed to find a way to re-execute
a class declaration[1]. The reason behind it was simple: the class
declaration in question had some external dependencies that could
potentially change during the lifetime of the program. I needed to be
able to execute the class 'declaration' block again to produce new
classes (in opposition to new instances, which would not solve my
problem)[2].

I solved the problem using inspect to find the current code object
*inside* the class declaration, and stored it an attribute of the
class itself. Then, a method of the class can be used to re-create it.
A metaclass-enabled version is shown below:

class Meta(type):
def __new__(klass, name, bases, dct):
dct['my_code'] = inspect.currentframe(1).f_code
return type(name, bases, dct)

class MyClass:
__metaclass__ = Meta
creation_time = time.ctime()
print "I'm inside my class definition!"
def re_execute(klass):
exec klass.my_code
return MyClass
re_execute = classmethod(re_execute)

Testing:

>>> class MyClass:

...
I'm inside my class definition!
>>> id(MyClass)

8737376
>>> MyClass.creation_time

'Tue Oct 05 21:50:25 2004'
>>> NewClass = MyClass.re_execute()

I'm inside my class definition!
>>> id(NewClass)

8736912
>>> NewClass.creation_time

'Tue Oct 05 21:51:20 2004'

Implementation notes:

1) The exec statement actually binds the name to the same original
name, but in the current local scope. That's why it does a 'return
MyClass' in the re_execute() method. The MyClass symbol refers to the
local symbol with this name that was just re-created by the exec
method.

2) If you call the exec from the same scope where the class was
originally created, then it is automatically bound to the same name:

>>> id(MyClass)

8737376
>>> MyClass.creation_time

'Tue Oct 05 21:50:25 2004'
>>> exec MyClass.my_code

I'm inside my class definition!
>>> id(MyClass)

8744512
>>> MyClass.creation_time

'Tue Oct 05 21:56:53 2004'

3) I thought that the same technique could be made to work without a
metaclass, but I couldn't make it work. I haven't invested much time
to understand why -- it just works as implemented above. (It seems
that it does not bind correctly to the original name as it does if you
use the metaclass)

----
[1] After a long and interesting discussion for which I have to thank
Alex Martelli and several other members of c.l.py, I've come to
realize that there are no real class declarations in Python. But for
all effects, it's just easier to call them this way, even if the
underlying mechanism is different from the one used on other
traditional languages.

[2] Think about it as a templating mechanism, similar to the C++ one,
but for Python; this mechanism generates new classes that can be
instantiated as many times as needed. It's not the same thing as to
customize a instance during its initialization.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: http://www.velocityreviews.com/forums/(E-Mail Removed)
mail: (E-Mail Removed)
 
Reply With Quote
 
 
 
 
Michele Simionato
Guest
Posts: n/a
 
      10-06-2004
Carlos Ribeiro <(E-Mail Removed)> wrote in message news:<(E-Mail Removed)>...
> As part of a pet project of mine, I needed to find a way to re-execute
> a class declaration[1].


You may want to read this old thread:

http://groups.google.com/groups?hl=e....lang.python.*

which does something close to what you ask for. Of course the trick
there is NOT recommended for real code

Michele Simionato
 
Reply With Quote
 
 
 
 
Alex Martelli
Guest
Posts: n/a
 
      10-06-2004
Carlos Ribeiro <(E-Mail Removed)> wrote:

> As part of a pet project of mine, I needed to find a way to re-execute
> a class declaration[1]. The reason behind it was simple: the class


If you start calling it 'class statement' I may read and perhaps help
with the rest of your post, but since I consider calling 'declaration'
something that is no more and no less than an executable statement an
excellent way to try and confuse the issues (by leading many readers to
expect something completely different from the reality), I'm going to
personally boycott any post deliberately perpetrating such wilful use of
language to confuse readers, and I urge the same course of action on
anybody who cares about clarity, understanding and communication.


Alex
 
Reply With Quote
 
Steve Holden
Guest
Posts: n/a
 
      10-06-2004
Alex Martelli wrote:

> Carlos Ribeiro <(E-Mail Removed)> wrote:
>
>
>>As part of a pet project of mine, I needed to find a way to re-execute
>>a class declaration[1]. The reason behind it was simple: the class

>
>
> If you start calling it 'class statement' I may read and perhaps help
> with the rest of your post, but since I consider calling 'declaration'
> something that is no more and no less than an executable statement an
> excellent way to try and confuse the issues (by leading many readers to
> expect something completely different from the reality), I'm going to
> personally boycott any post deliberately perpetrating such wilful use of
> language to confuse readers, and I urge the same course of action on
> anybody who cares about clarity, understanding and communication.
>
>
> Alex


How would you feel about "class definition"?

regards
Steve
--
http://www.holdenweb.com
http://pydish.holdenweb.com
Holden Web LLC +1 800 494 3119
 
Reply With Quote
 
Alex Martelli
Guest
Posts: n/a
 
      10-06-2004
Steve Holden <(E-Mail Removed)> wrote:
...
> >>As part of a pet project of mine, I needed to find a way to re-execute
> >>a class declaration[1]. The reason behind it was simple: the class

...
> > If you start calling it 'class statement' I may read and perhaps help
> > with the rest of your post, but since I consider calling 'declaration'

...
> How would you feel about "class definition"?


Since we have a 'def' statement it seems confusing to me to talk of
'definition' about something else than a 'def' (even there I prefer "a
def statement" as clearer and more direct, but I guess I'll have to
accept "a function definition" given ``def''... I do wish that keyword
was 'function' rather than 'def', but, oh well, too late for that.


Alex
 
Reply With Quote
 
Carlos Ribeiro
Guest
Posts: n/a
 
      10-06-2004
In respectful consideration to Alex's opposition to the misuse of the
word 'declaration', here is a repost with the suggested changes:
---
As part of a pet project of mine, I needed to find a way to re-execute
a class statement. The reason behind it was simple: the class
statement in question had some external dependencies that could
potentially change during the lifetime of the program. I needed to be
able to execute the class statement block again to produce new
classes (in opposition to new instances, which would not solve my
problem)[1].

I solved the problem using inspect to find the current code object
*inside* the class statement, and stored it in an attribute of the
class itself. Then, a method of the class can be used to re-create it.
A metaclass-enabled version is shown below:

class Meta(type):
def __new__(klass, name, bases, dct):
dct['my_code'] = inspect.currentframe(1).f_code
return type(name, bases, dct)

class MyClass:
__metaclass__ = Meta
creation_time = time.ctime()
print "I'm inside my class definition!"
def re_execute(klass):
exec klass.my_code
return MyClass
re_execute = classmethod(re_execute)

Testing:

>>> class MyClass:

...
I'm inside my class definition!
>>> id(MyClass)

8737376
>>> MyClass.creation_time

'Tue Oct 05 21:50:25 2004'
>>> NewClass = MyClass.re_execute()

I'm inside my class definition!
>>> id(NewClass)

8736912
>>> NewClass.creation_time

'Tue Oct 05 21:51:20 2004'

Implementation notes:

1) The exec statement actually binds the name to the same original
name, but in the current local scope. That's why it does a 'return
MyClass' in the re_execute() method. The MyClass symbol refers to the
local symbol with this name that was just re-created by the exec
method.

2) If you call the exec from the same scope where the class was
originally created, then it is automatically bound to the same name:

>>> id(MyClass)

8737376
>>> MyClass.creation_time

'Tue Oct 05 21:50:25 2004'
>>> exec MyClass.my_code

I'm inside my class definition!
>>> id(MyClass)

8744512
>>> MyClass.creation_time

'Tue Oct 05 21:56:53 2004'

3) I thought that the same technique could be made to work without a
metaclass, but I couldn't make it work. I haven't invested much time
to understand why -- it just works as implemented above. (It seems
that it does not bind correctly to the original name as it does if you
use the metaclass)

----
[1] Think about it as a templating mechanism, similar to the C++ one,
but for Python; this mechanism generates new classes that can be
instantiated as many times as needed. It's not the same thing as to
customize a instance during its initialization.

--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (E-Mail Removed)
mail: (E-Mail Removed)
 
Reply With Quote
 
Alex Martelli
Guest
Posts: n/a
 
      10-06-2004
Carlos Ribeiro <(E-Mail Removed)> wrote:
...
> As part of a pet project of mine, I needed to find a way to re-execute
> a class statement. The reason behind it was simple: the class
> statement in question had some external dependencies that could
> potentially change during the lifetime of the program. I needed to be
> able to execute the class statement block again to produce new
> classes (in opposition to new instances, which would not solve my
> problem)[1].


Interesting problem.


> I solved the problem using inspect to find the current code object
> *inside* the class statement, and stored it in an attribute of the
> class itself. Then, a method of the class can be used to re-create it.
> A metaclass-enabled version is shown below:
>
> class Meta(type):
> def __new__(klass, name, bases, dct):
> dct['my_code'] = inspect.currentframe(1).f_code
> return type(name, bases, dct)
>
> class MyClass:
> __metaclass__ = Meta
> creation_time = time.ctime()
> print "I'm inside my class definition!"
> def re_execute(klass):
> exec klass.my_code
> return MyClass
> re_execute = classmethod(re_execute)


Urk. Ahem, I mean, interesting solution. I _think_ it poses certain,
ahem, problems, such as infinite recursion if a call to re_execute is in
the same code object as that where the 'class MyClass' is, for example
-- because basically you ARE re-executing the whole kaboodle. You're
not just re-executing the 'class' statement, but everything that comes
before or after it in the same containing codeobject. Your tests in an
interactive interpreter are tricky that way, because in that special
environment a code object is a tiny little thing. But stick this into a
module and place a dis.dis(dct['my_code']) in the metaclass's __new__
and you'll see that basically MyClass.re_execute isn't very different
from a reload of the module. Isn't that a _bit_ too much in general?

The code object for the body of the class can also be found (with some
work): it will be among the co_consts of what you've stored as
dct['my_code'], it will be an instance of types.CodeType, and its
co_name will be the same as the 'name' parameter to Meta.__new__. If
you have more than one code object with those characteristics, e.g. two
classes with the same name at the same module's toplevel, you do have
some trouble. But otherwise, you can build a function, with
new.function(thatclassbodycodeobject, theglobalsfrominspect), which,
when called (w/o arguments), reexecutes the classbody and returns the
same kind of dictionary that Meta.__new__ is receiving as its 'dct'
argument. Wouldn't this be preferable to re-executing the whole module
body, perhaps?

> Implementation notes:
>
> 1) The exec statement actually binds the name to the same original
> name, but in the current local scope. That's why it does a 'return
> MyClass' in the re_execute() method. The MyClass symbol refers to the
> local symbol with this name that was just re-created by the exec
> method.


This also fails if you have multiple uses of name MyClass at toplevel of
the same module -- it actually fails more often than my suggestions
above, e.g if your module is:

class MyClass:
... etc ...
FakeName = MyClass
MyClass = 23

then your FakeName.re_execute() will return 23, while the approach I
suggest will be able to recover in this case (the codeobject in the
co_consts of the module's code object will still be the only one in that
container that's named 'MyClass').


> 2) If you call the exec from the same scope where the class was
> originally created, then it is automatically bound to the same name:


Same issue as above -- you're getting closer and closer to a module's
reload (save in the anomalous case where you type the whole class
statement at an interactive prompt. Note that it IS quite possible
that what you want to do IS in fact to reload the module, period, but,
if so, then the reload statement might be handier.

> 3) I thought that the same technique could be made to work without a
> metaclass, but I couldn't make it work. I haven't invested much time
> to understand why -- it just works as implemented above. (It seems
> that it does not bind correctly to the original name as it does if you
> use the metaclass)


It appears to me, but I haven't tested this, that putting the inspect
introspection in the classbody itself should be a sounder technique, as
it should be able to find itself and stash itself away more reliably
than just based on name (which is the best the metaclass can do). The
name binding issue is separate, however.

> [1] Think about it as a templating mechanism, similar to the C++ one,
> but for Python; this mechanism generates new classes that can be
> instantiated as many times as needed. It's not the same thing as to
> customize a instance during its initialization.


I agree, there's one metalevel of difference between instantiating a
class to make an instance, and instantiating a metaclass to make a
class.


Alex
 
Reply With Quote
 
Carlos Ribeiro
Guest
Posts: n/a
 
      10-06-2004
On Wed, 6 Oct 2004 14:06:23 +0200, Alex Martelli <(E-Mail Removed)> wrote:
> Urk. Ahem, I mean, interesting solution. I _think_ it poses certain,
> ahem, problems, such as infinite recursion if a call to re_execute is in
> the same code object as that where the 'class MyClass' is, for example
> -- because basically you ARE re-executing the whole kaboodle. You're
> not just re-executing the 'class' statement, but everything that comes
> before or after it in the same containing codeobject. Your tests in an
> interactive interpreter are tricky that way, because in that special
> environment a code object is a tiny little thing. But stick this into a
> module and place a dis.dis(dct['my_code']) in the metaclass's __new__
> and you'll see that basically MyClass.re_execute isn't very different
> from a reload of the module. Isn't that a _bit_ too much in general?


(repeating to myself: dis is your friend. dis is your friend).

Well, it seems that I'm running out of options. I've checked dis, and
done a few more tests. I found a way to find the correct code object
for the _body_ of the class statement (or at least I think so), but
this does not solve the problem, as it will be shown below:

-------------
import inspect
import dis
import time
import types

class Meta(type):
def __new__(klass, name, bases, dct):
caller = inspect.stack(0)[1]
caller_code = caller[0].f_code
caller_filename = caller[1]
caller_lineno = caller[2]
print "Caller info: ", caller_filename, caller_lineno, caller_code
print dis.dis(caller[0].f_code)
for code in caller_code.co_consts:
if isinstance(code, types.CodeType):
if (code.co_filename == caller_filename) and \
(code.co_firstlineno == caller_lineno):
print "found!", code.co_name
dct['my_code'] = code
return type(name, bases, dct)

class MyClass(object):
__metaclass__ = Meta
creation_time = time.ctime()
print "I'm inside my class definition!"
def re_execute(klass):
exec klass.my_code
return MyClass
re_execute = classmethod(re_execute)
-------------

The code checks the filename and the line number to find the correct
code object. It could also check the class name, just to make sure
that everything matches. It isn't needed, as far as I know. But the
problem is that the code block that is found refers to the class
_body_ only. Checking dis.dis() helps to understand it:

27 64 LOAD_CONST 4 ('MyClass')
67 BUILD_TUPLE 0
70 LOAD_CONST 5 (<code object MyClass at
01209E60, file "c:\work\help-on-c-l-py\re_exec_class.py", line 27>)
73 MAKE_FUNCTION 0
76 CALL_FUNCTION 0
79 BUILD_CLASS
80 STORE_NAME 7 (MyClass)

There is not a single code block that does all the tasks related to
the class statement, and only it. In fact, there are two different
code objects to choose from, and neither really solves the problem.
One is the module object; as you've pointed out, it's too generic and
executes too much stuff, not just the class statement. The other one
is the <code object MyClass at 01209E60> referred to in the dis()
output above, that is the _body_ of the class statement. Calling it
doesn't call the metaclass, and neither does the name binding 'magic':

>>> exec MyClass.my_code

I'm inside my class definition!
# it does not print any of the debug messages in the metaclass, and
it's obvious why

It also doesn't take into account that the class may have base
classes. For example, changing the statement to 'class
MyClass(object)', the compiled code looks like this:

27 64 LOAD_CONST 4 ('MyClass')
67 LOAD_NAME 7 (object)
70 BUILD_TUPLE 1

>From this point, few options are left. What I need to execute is in

fact a part of the module code object -- just one line of it, that's
the code that wraps the entire class statement. Besides giving up
(which is a reasonable option, given the constraints), what is left
are some horrible hacks. (I'm don't feel like discussing it here; it's
like Mordor Black Speech, if shall not be uttered in a public forum
.

BTW, the two code objects represent exactly the two alternative
solutions that can be done using only pure Python code:

1) reload the entire module (which is nearly equivalent to calling the
module code object, with the difference that it doesn't reload and
recompile the actual source code)

2) enclose the class statement inside a def, and run the def to obtain
the class template:

def BuildTemplate(<template parameters>):
class MyTemplate(...):
...
return MyTemplate

For my application, I'm tending towards (2), because it allows for a
better modularization of the code. But I'm sorry, in a way, because it
breaks the expect semantics in a way -- it's a function that returns a
class. It's valid, but some may regard it as 'ugly' and 'unpythonic'.

> > 3) I thought that the same technique could be made to work without a
> > metaclass, but I couldn't make it work. I haven't invested much time
> > to understand why -- it just works as implemented above. (It seems
> > that it does not bind correctly to the original name as it does if you
> > use the metaclass)

>
> It appears to me, but I haven't tested this, that putting the inspect
> introspection in the classbody itself should be a sounder technique, as
> it should be able to find itself and stash itself away more reliably
> than just based on name (which is the best the metaclass can do). The
> name binding issue is separate, however.


I've tried it yesterday, and it won't work, for some still unknown
reason. I could find the code object, but when I ran it, it would not
bind to the name in the same way as the metaclass would do. But it
doesn't matter at this point.

> > [1] Think about it as a templating mechanism, similar to the C++ one,
> > but for Python; this mechanism generates new classes that can be
> > instantiated as many times as needed. It's not the same thing as to
> > customize a instance during its initialization.

>
> I agree, there's one metalevel of difference between instantiating a
> class to make an instance, and instantiating a metaclass to make a
> class.



--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (E-Mail Removed)
mail: (E-Mail Removed)
 
Reply With Quote
 
Carlos Ribeiro
Guest
Posts: n/a
 
      10-06-2004
On 5 Oct 2004 23:33:01 -0700, Michele Simionato
<(E-Mail Removed)> wrote:
> which does something close to what you ask for. Of course the trick
> there is NOT recommended for real code


That was the best advice I could have Really. I'm trying to outwit
Python, and one has to have the good sense to stop doing it at some
point. I'm giving up (unless I found a really great way to make it),
and I'll use this idiom instead:

def BuildTemplate(<template parameters>)
class TemplateClass(...):
...
return TemplateClass

It's simple, and less hackish, but still unusual. I fear I may be
missing something on my design, if I need to resort to such hacks.
--
Carlos Ribeiro
Consultoria em Projetos
blog: http://rascunhosrotos.blogspot.com
blog: http://pythonnotes.blogspot.com
mail: (E-Mail Removed)
mail: (E-Mail Removed)
 
Reply With Quote
 
Alex Martelli
Guest
Posts: n/a
 
      10-06-2004
Carlos Ribeiro <(E-Mail Removed)> wrote:
...
> > module and place a dis.dis(dct['my_code']) in the metaclass's __new__
> > and you'll see that basically MyClass.re_execute isn't very different
> > from a reload of the module. Isn't that a _bit_ too much in general?

>
> (repeating to myself: dis is your friend. dis is your friend).


If you're hacking at this level, it sure is.

> done a few more tests. I found a way to find the correct code object
> for the _body_ of the class statement (or at least I think so), but


Good -- as I said I think it can be even more reliably found from inside
the classbody itself, but, no big deal.

> exec klass.my_code


Ah, this is your problem -- that's not what you want to do now.

> problem is that the code block that is found refers to the class
> _body_ only. Checking dis.dis() helps to understand it:


Exactly: it's what builds the dict you want to pass to the metaclass
(once you wrap it into a new.function, as I mentioned).

> There is not a single code block that does all the tasks related to
> the class statement, and only it. In fact, there are two different
> code objects to choose from, and neither really solves the problem.


Not by itself, but add some code and you'll be fine.

1. You need to build an f = new.function(klass.my_code, globals()) in
your classmethod that now erroneously does the exec above quoted (you
could in fact do this in the metaclass, save the new function rather
than the code object -- get the right globals via inspect).

2. you need to obtain the needed dict, that's just d = f() in your
method

3. you can now call your metaclass with the appropriate bases and name:
klas = self.__class__ # or cls if this is a classmethod
return type(klas)(klas.__name__, klas.__bases__, d)

voila, you're there. This is all coded to minimize the need to have
this method located in a specific class -- as I said I think you
probably want to have this stuff in the metaclass (if you have a
metaclass at all), but, whatever.


This is basically just mimicking what Python itself does (check the
dis.dis for a class statement as a starter): get the class body's code
object, build a function from it, call the function (it returns its
locals() as the result -- that 'return' is of course in the class body's
code object, and dis.dis shows that...), and that's how you get the
classdict on the stack. Add name and bases, call the metaclass, and
Bob's your uncle...


Alex
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
While executing the class definition which object is referenced bythe first argument of the class method, Y r Object attributes not allowed asdefault arguments Krishna Python 4 03-07-2008 09:44 PM
Why 'class spam(object)' instead of class spam(Object)' ? Sergio Correia Python 7 09-18-2007 02:07 AM
Nested Class, Member Class, Inner Class, Local Class, Anonymous Class E11 Java 1 10-12-2005 03:34 PM
Object creation - Do we really need to create a parent for a derieved object - can't the base object just point to an already created base object jon wayne C++ 9 09-22-2005 02:06 AM
Passing derived class object array in place of base class object array justanotherguy63@yahoo.com C++ 9 12-03-2004 10:57 PM



Advertisments