Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Overriding a method at the instance level on a subclass of a builtintype

Reply
Thread Tools

Overriding a method at the instance level on a subclass of a builtintype

 
 
Zac Burns
Guest
Posts: n/a
 
      12-03-2008
Sorry for the long subject.

I'm trying to create a subclass dictionary that runs extra init code
on the first __getitem__ call. However, the performance of __getitem__
is quite important - so I'm trying in the subclassed __getitem__
method to first run some code and then patch in the original dict
method for the instance to avoid even the check to see if the init
code has been run. Various recipes using instancemethod and the like
have failed me.

Curiously if __slots__ is not specified no error occurs when setting
self.__getitem__ but the function is not overriden. If __slots__ is
['__getitem__'] however it complains that __getitem__ is read only. I
do not understand that behavior.

--
Zachary Burns
(407)590-4814
Aim - Zac256FL
Production Engineer (Digital Overlord)
Zindagi Games
 
Reply With Quote
 
 
 
 
George Sakkis
Guest
Posts: n/a
 
      12-03-2008
On Dec 2, 7:58*pm, "Zac Burns" <(E-Mail Removed)> wrote:

> Sorry for the long subject.
>
> I'm trying to create a subclass dictionary that runs extra init code
> on the first __getitem__ call. However, the performance of __getitem__
> is quite important - so I'm trying in the subclassed __getitem__
> method to first run some code and then patch in the original dict
> method for the instance to avoid even the check to see if the init
> code has been run. Various recipes using instancemethod and the like
> have failed me.


For new-style classes, special methods are always looked up in the
class, not the instance, so you're out of luck there. What are you
trying to do? Perhaps there is a less magic solution to the general
problem.

George
 
Reply With Quote
 
 
 
 
Aaron Brady
Guest
Posts: n/a
 
      12-03-2008
On Dec 2, 6:58*pm, "Zac Burns" <(E-Mail Removed)> wrote:
> Sorry for the long subject.
>
> I'm trying to create a subclass dictionary that runs extra init code
> on the first __getitem__ call. However, the performance of __getitem__
> is quite important - so I'm trying in the subclassed __getitem__
> method to first run some code and then patch in the original dict
> method for the instance to avoid even the check to see if the init
> code has been run. Various recipes using instancemethod and the like
> have failed me.
>
> Curiously if __slots__ is not specified no error occurs when setting
> self.__getitem__ but the function is not overriden. If __slots__ is
> ['__getitem__'] however it complains that __getitem__ is read only. I
> do not understand that behavior.
>
> --
> Zachary Burns
> (407)590-4814
> Aim - Zac256FL
> Production Engineer (Digital Overlord)
> Zindagi Games


That sounds like the State Pattern, from GoF. http://en.wikipedia.org/wiki/State_pattern

I like the idea of 'renaming', not redefining, but reassigning methods
at different points during an object's lifetime. I often wish I had
more experience with it, and more docs talked about it.

It's hard on memory usage, since each instance has its own function
attribute, even if there's still only one instance of the function.
Without it, the function attribute is just looked up on the class.

Not thoroughly tested:

>>> class A:

.... def methA( self ):
.... print 'methA'
.... self.meth= self.methB
.... meth= methA
.... def methB( self ):
.... print 'methB'
....
>>> a= A()
>>> a.meth()

methA
>>> a.meth()

methB
 
Reply With Quote
 
Bryan Olson
Guest
Posts: n/a
 
      12-03-2008
Zac Burns wrote:
> Sorry for the long subject.
>
> I'm trying to create a subclass dictionary that runs extra init code
> on the first __getitem__ call. However, the performance of __getitem__
> is quite important - so I'm trying in the subclassed __getitem__
> method to first run some code and then patch in the original dict
> method for the instance to avoid even the check to see if the init
> code has been run. Various recipes using instancemethod and the like
> have failed me.


One option is to re-assign the object's __class__, as in:

class XDict (dict):
pass

class ZDict (XDict):
def __getitem__(self, k):
whatever_you_want_to_do_once(self)
result = dict.__getitem__(self, k)
self.__class__ = XDict
return result


The first dict subtype is needed because __class__ assignment requires
that both the current and newly-assigned class be 'heap types', which
the native dict is not.


--
--Bryan
 
Reply With Quote
 
Arnaud Delobelle
Guest
Posts: n/a
 
      12-03-2008
"Zac Burns" <(E-Mail Removed)> writes:

> Sorry for the long subject.
>
> I'm trying to create a subclass dictionary that runs extra init code
> on the first __getitem__ call. However, the performance of __getitem__
> is quite important - so I'm trying in the subclassed __getitem__
> method to first run some code and then patch in the original dict
> method for the instance to avoid even the check to see if the init
> code has been run. Various recipes using instancemethod and the like
> have failed me.
>
> Curiously if __slots__ is not specified no error occurs when setting
> self.__getitem__ but the function is not overriden. If __slots__ is
> ['__getitem__'] however it complains that __getitem__ is read only. I
> do not understand that behavior.


You can change the class on the fly to achieve what you want:

>>> class D1(dict):

.... def __getitem__(self, key):
.... print 'first call'
.... self.__class__ = D2
.... return dict.__getitem__(self, key)
....
>>> class D2(dict):

.... pass
....
>>> d = D1(foo=42)
>>> d['foo']

first call
42
>>> d['foo']

42
>>>


HTH

--
Arnaud
 
Reply With Quote
 
Jason Scheirer
Guest
Posts: n/a
 
      12-03-2008
On Dec 2, 6:13*pm, Aaron Brady <(E-Mail Removed)> wrote:
> On Dec 2, 6:58*pm, "Zac Burns" <(E-Mail Removed)> wrote:
>
>
>
> > Sorry for the long subject.

>
> > I'm trying to create a subclass dictionary that runs extra init code
> > on the first __getitem__ call. However, the performance of __getitem__
> > is quite important - so I'm trying in the subclassed __getitem__
> > method to first run some code and then patch in the original dict
> > method for the instance to avoid even the check to see if the init
> > code has been run. Various recipes using instancemethod and the like
> > have failed me.

>
> > Curiously if __slots__ is not specified no error occurs when setting
> > self.__getitem__ but the function is not overriden. If __slots__ is
> > ['__getitem__'] however it complains that __getitem__ is read only. I
> > do not understand that behavior.

>
> > --
> > Zachary Burns
> > (407)590-4814
> > Aim - Zac256FL
> > Production Engineer (Digital Overlord)
> > Zindagi Games

>
> That sounds like the State Pattern, from GoF. *http://en.wikipedia.org/wiki/State_pattern
>
> I like the idea of 'renaming', not redefining, but reassigning methods
> at different points during an object's lifetime. *I often wish I had
> more experience with it, and more docs talked about it.
>
> It's hard on memory usage, since each instance has its own function
> attribute, even if there's still only one instance of the function.
> Without it, the function attribute is just looked up on the class.
>
> Not thoroughly tested:
>
> >>> class A:

>
> ... * * def methA( self ):
> ... * * * * * * print 'methA'
> ... * * * * * * self.meth= self.methB
> ... * * meth= methA
> ... * * def methB( self ):
> ... * * * * * * print 'methB'
> ...>>> a= A()
> >>> a.meth()

> methA
> >>> a.meth()

>
> methB


The problem with using this this pattern in the way that you've
specified is that you have a potential memory leak/object lifetime
issue. Assigning a bound method of an instance (which itself holds a
reference to self) to another attribute in that same instance creates
a kind of circular dependency that I have discovered can trip up the
GC more often than not.

You can subclass it as easily:

class dictsubclass(dict):
def __getitem__(self, keyname):
if not hasattr(self, '_run_once'):
self.special_code_to_run_once()
self._run_once = True
return super(self, dict).__getitem__(keyname)

If that extra ~16 bytes associated with the subclass is really a
problem:

class dictsubclass(dict):
def __getitem__(self, keyname):
self.special_code_to_run_once()
self.__class__ = dict
return super(self, dict).__getitem__(keyname)

But I don't think that's a good idea at all.
 
Reply With Quote
 
Aaron Brady
Guest
Posts: n/a
 
      12-04-2008
On Dec 3, 1:25*pm, Jason Scheirer <(E-Mail Removed)> wrote:
> On Dec 2, 6:13*pm, Aaron Brady <(E-Mail Removed)> wrote:
> > >>> class A:

>
> > ... * * def methA( self ):
> > ... * * * * * * print 'methA'
> > ... * * * * * * self.meth= self.methB
> > ... * * meth= methA
> > ... * * def methB( self ):
> > ... * * * * * * print 'methB'
> > ...>>> a= A()
> > >>> a.meth()

> > methA
> > >>> a.meth()

>
> > methB

>
> The problem with using this this pattern in the way that you've
> specified is that you have a potential memory leak/object lifetime
> issue. Assigning a bound method of an instance (which itself holds a
> reference to self) to another attribute in that same instance creates
> a kind of circular dependency that I have discovered can trip up the
> GC more often than not.
>
> You can subclass it as easily:
>
> class dictsubclass(dict):
> * * def __getitem__(self, keyname):
> * * * * if not hasattr(self, '_run_once'):
> * * * * * * self.special_code_to_run_once()
> * * * * * * self._run_once = True
> * * * * return super(self, dict).__getitem__(keyname)
>
> If that extra ~16 bytes associated with the subclass is really a
> problem:
>
> class dictsubclass(dict):
> * * def __getitem__(self, keyname):
> * * * * self.special_code_to_run_once()
> * * * * self.__class__ = dict
> * * * * return super(self, dict).__getitem__(keyname)
>
> But I don't think that's a good idea at all.


Interesting. The following code ran, and process memory usage rose to
150MB. It failed to return to normal afterward.

>>> for x in range( 10000000 ):

.... a= []
.... a.append(a)
....

However, the following code succeeded in returning usage to normal.

>>> import gc
>>> gc.collect()


It was in version 2.6. So, the GC succeeded in collecting circularly
linked garbage when invoked manually. That might have implications in
the OP's use case.

In another language, it would work differently, if it lacked unbound
method descriptors. C++ for example, untested:

class C {
public:
func_t meth;
C( ) { meth= methA; }
void methA( ) { meth= methB; }
void methB( ) { }
};

It has no problems with memory consumption (an extra pointer per
object), or circular references; functions are not first-class
objects. However they are in Python, which creates an entire bound
method object per instance.

The OP stated:

> run some code and then patch in the original dict
> method for the instance to avoid even the check to see if the init
> code has been run.


So your, Arnaud's, and Bryan's '.__class__' solution is probably best,
and possibly even truer to the intent of the State Pattern.

It is too bad that you can't assign an unbound method to the member,
and derive the bound method on the fly. That might provide a middle-
ground solution.


 
Reply With Quote
 
Zac Burns
Guest
Posts: n/a
 
      12-04-2008
The class method seems to be the most promising, however I have more
'state' methods to worry about so I might end up building new classes
on the fly rather than have a class per permutation of states! Now the
code isn't quite as clear as I thought it was going to be.

It seems unfortunate to me that methods are always looked up on the
class for new style objects. Was this done for speed reasons?
--
Zachary Burns
(407)590-4814
Aim - Zac256FL
Production Engineer (Digital Overlord)
Zindagi Games



On Thu, Dec 4, 2008 at 2:38 AM, Aaron Brady <(E-Mail Removed)> wrote:
> On Dec 3, 1:25 pm, Jason Scheirer <(E-Mail Removed)> wrote:
>> On Dec 2, 6:13 pm, Aaron Brady <(E-Mail Removed)> wrote:
>> > >>> class A:

>>
>> > ... def methA( self ):
>> > ... print 'methA'
>> > ... self.meth= self.methB
>> > ... meth= methA
>> > ... def methB( self ):
>> > ... print 'methB'
>> > ...>>> a= A()
>> > >>> a.meth()
>> > methA
>> > >>> a.meth()

>>
>> > methB

>>
>> The problem with using this this pattern in the way that you've
>> specified is that you have a potential memory leak/object lifetime
>> issue. Assigning a bound method of an instance (which itself holds a
>> reference to self) to another attribute in that same instance creates
>> a kind of circular dependency that I have discovered can trip up the
>> GC more often than not.
>>
>> You can subclass it as easily:
>>
>> class dictsubclass(dict):
>> def __getitem__(self, keyname):
>> if not hasattr(self, '_run_once'):
>> self.special_code_to_run_once()
>> self._run_once = True
>> return super(self, dict).__getitem__(keyname)
>>
>> If that extra ~16 bytes associated with the subclass is really a
>> problem:
>>
>> class dictsubclass(dict):
>> def __getitem__(self, keyname):
>> self.special_code_to_run_once()
>> self.__class__ = dict
>> return super(self, dict).__getitem__(keyname)
>>
>> But I don't think that's a good idea at all.

>
> Interesting. The following code ran, and process memory usage rose to
> 150MB. It failed to return to normal afterward.
>
>>>> for x in range( 10000000 ):

> ... a= []
> ... a.append(a)
> ...
>
> However, the following code succeeded in returning usage to normal.
>
>>>> import gc
>>>> gc.collect()

>
> It was in version 2.6. So, the GC succeeded in collecting circularly
> linked garbage when invoked manually. That might have implications in
> the OP's use case.
>
> In another language, it would work differently, if it lacked unbound
> method descriptors. C++ for example, untested:
>
> class C {
> public:
> func_t meth;
> C( ) { meth= methA; }
> void methA( ) { meth= methB; }
> void methB( ) { }
> };
>
> It has no problems with memory consumption (an extra pointer per
> object), or circular references; functions are not first-class
> objects. However they are in Python, which creates an entire bound
> method object per instance.
>
> The OP stated:
>
>> run some code and then patch in the original dict
>> method for the instance to avoid even the check to see if the init
>> code has been run.

>
> So your, Arnaud's, and Bryan's '.__class__' solution is probably best,
> and possibly even truer to the intent of the State Pattern.
>
> It is too bad that you can't assign an unbound method to the member,
> and derive the bound method on the fly. That might provide a middle-
> ground solution.
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>

 
Reply With Quote
 
Arnaud Delobelle
Guest
Posts: n/a
 
      12-04-2008
"Zac Burns" <(E-Mail Removed)> writes:

> The class method seems to be the most promising, however I have more
> 'state' methods to worry about so I might end up building new classes
> on the fly rather than have a class per permutation of states! Now the
> code isn't quite as clear as I thought it was going to be.
>
> It seems unfortunate to me that methods are always looked up on the
> class for new style objects. Was this done for speed reasons?


It's only special methods such as __getitem__, ...

You can override normal method on a per-object basis just by adding a
callable attribute with its name to the object:

>>> class A(object):

.... def foo(self): print 'A.foo'
....
>>> a = A()
>>> a.foo()

A.foo
>>> def a_foo(): print 'a.foo'

....
>>> a.foo = a_foo
>>> a.foo()

a.foo

--
Arnaud
 
Reply With Quote
 
George Sakkis
Guest
Posts: n/a
 
      12-04-2008
On Dec 4, 12:31*pm, Arnaud Delobelle <(E-Mail Removed)> wrote:
> "Zac Burns" <(E-Mail Removed)> writes:
> > The class method seems to be the most promising, however I have more
> > 'state' methods to worry about so I might end up building new classes
> > on the fly rather than have a class per permutation of states! Now the
> > code isn't quite as clear as I thought it was going to be.

>
> > It seems unfortunate to me that methods are always looked up on the
> > class for new style objects. Was this done for speed reasons?

>
> It's only special methods such as __getitem__, ...
>
> You can override normal method on a per-object basis just by adding a
> callable attribute with its name to the object:
>
> >>> class A(object):

>
> ... * * def foo(self): print 'A.foo'
> ...>>> a = A()
> >>> a.foo()

> A.foo
> >>> def a_foo(): print 'a.foo'

> ...
> >>> a.foo = a_foo
> >>> a.foo()


Note that the overriden "method" here is a plain function; it doesn't
take self as the first argument. If you want to bind it to a callable
that expects the first argument to be self, you have to bind
explicitly self to the object:

>>> def a_foo(self): print 'a.foo'
>>> a.foo = a_foo
>>> a.foo()

TypeError: a_foo() takes exactly 1 argument (0 given)
>>> from functools import partial
>>> a.foo = partial(a_foo,a)
>>> a.foo()

a_foo

George
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
how to stop the subclass from overriding a method. Venkat Akkineni Ruby 11 08-01-2009 12:35 PM
instance method adding another instance method to the class Raj Singh Ruby 2 05-29-2008 10:09 PM
Can you create an instance of a subclass with an existing instance of the base class? Sandra-24 Python 18 04-29-2006 04:01 PM
String subclass method returns subclass - bug or feature? S.Volkov Ruby 2 03-12-2006 06:46 PM
Problem when subclass instance changes base class instance variable Gerry Sutton Python 1 04-16-2005 06:06 AM



Advertisments