Overriding a method at the instance level on a subclass of a builtintype
 

Sorry for the long subject.

I'm trying to create a subclass dictionary that runs extra init code
on the first __getitem__ call. However, the performance of __getitem__
is quite important - so I'm trying in the subclassed __getitem__
method to first run some code and then patch in the original dict
method for the instance to avoid even the check to see if the init
code has been run. Various recipes using instancemethod and the like
have failed me.

Curiously if __slots__ is not specified no error occurs when setting
self.__getitem__ but the function is not overriden. If __slots__ is
['__getitem__'] however it complains that __getitem__ is read only. I
do not understand that behavior.

--
Zachary Burns
(407)590-4814
Aim - Zac256FL
Production Engineer (Digital Overlord)
Zindagi Games

Re: Overriding a method at the instance level on a subclass of abuiltin type
 

On Dec 2, 7:58*pm, "Zac Burns" <zac...@gmail.com> wrote:

> Sorry for the long subject.
>
> I'm trying to create a subclass dictionary that runs extra init code
> on the first __getitem__ call. However, the performance of __getitem__
> is quite important - so I'm trying in the subclassed __getitem__
> method to first run some code and then patch in the original dict
> method for the instance to avoid even the check to see if the init
> code has been run. Various recipes using instancemethod and the like
> have failed me.

For new-style classes, special methods are always looked up in the
class, not the instance, so you're out of luck there. What are you
trying to do? Perhaps there is a less magic solution to the general
problem.

George

Re: Overriding a method at the instance level on a subclass of abuiltin type
 

On Dec 2, 6:58*pm, "Zac Burns" <zac...@gmail.com> wrote:
> Sorry for the long subject.
>
> I'm trying to create a subclass dictionary that runs extra init code
> on the first __getitem__ call. However, the performance of __getitem__
> is quite important - so I'm trying in the subclassed __getitem__
> method to first run some code and then patch in the original dict
> method for the instance to avoid even the check to see if the init
> code has been run. Various recipes using instancemethod and the like
> have failed me.
>
> Curiously if __slots__ is not specified no error occurs when setting
> self.__getitem__ but the function is not overriden. If __slots__ is
> ['__getitem__'] however it complains that __getitem__ is read only. I
> do not understand that behavior.
>
> --
> Zachary Burns
> (407)590-4814
> Aim - Zac256FL
> Production Engineer (Digital Overlord)
> Zindagi Games

That sounds like the State Pattern, from GoF. http://en.wikipedia.org/wiki/State_pattern

I like the idea of 'renaming', not redefining, but reassigning methods
at different points during an object's lifetime. I often wish I had
more experience with it, and more docs talked about it.

It's hard on memory usage, since each instance has its own function
attribute, even if there's still only one instance of the function.
Without it, the function attribute is just looked up on the class.

Not thoroughly tested:

>>> class A:
.... def methA( self ):
.... print 'methA'
.... self.meth= self.methB
.... meth= methA
.... def methB( self ):
.... print 'methB'
....
>>> a= A()
>>> a.meth()
methA
>>> a.meth()
methB

Re: Overriding a method at the instance level on a subclass of abuiltin type
 

Zac Burns wrote:
> Sorry for the long subject.
>
> I'm trying to create a subclass dictionary that runs extra init code
> on the first __getitem__ call. However, the performance of __getitem__
> is quite important - so I'm trying in the subclassed __getitem__
> method to first run some code and then patch in the original dict
> method for the instance to avoid even the check to see if the init
> code has been run. Various recipes using instancemethod and the like
> have failed me.

One option is to re-assign the object's __class__, as in:

class XDict (dict):
pass

class ZDict (XDict):
def __getitem__(self, k):
whatever_you_want_to_do_once(self)
result = dict.__getitem__(self, k)
self.__class__ = XDict
return result


The first dict subtype is needed because __class__ assignment requires
that both the current and newly-assigned class be 'heap types', which
the native dict is not.


--
--Bryan

Re: Overriding a method at the instance level on a subclass of a builtin type
 

"Zac Burns" <zac256@gmail.com> writes:

> Sorry for the long subject.
>
> I'm trying to create a subclass dictionary that runs extra init code
> on the first __getitem__ call. However, the performance of __getitem__
> is quite important - so I'm trying in the subclassed __getitem__
> method to first run some code and then patch in the original dict
> method for the instance to avoid even the check to see if the init
> code has been run. Various recipes using instancemethod and the like
> have failed me.
>
> Curiously if __slots__ is not specified no error occurs when setting
> self.__getitem__ but the function is not overriden. If __slots__ is
> ['__getitem__'] however it complains that __getitem__ is read only. I
> do not understand that behavior.

You can change the class on the fly to achieve what you want:

>>> class D1(dict):
.... def __getitem__(self, key):
.... print 'first call'
.... self.__class__ = D2
.... return dict.__getitem__(self, key)
....
>>> class D2(dict):
.... pass
....
>>> d = D1(foo=42)
>>> d['foo']
first call
42
>>> d['foo']
42
>>>

HTH

--
Arnaud

Re: Overriding a method at the instance level on a subclass of abuiltin type
 

On Dec 2, 6:13*pm, Aaron Brady <castiro...@gmail.com> wrote:
> On Dec 2, 6:58*pm, "Zac Burns" <zac...@gmail.com> wrote:
>
>
>
> > Sorry for the long subject.
>
> > I'm trying to create a subclass dictionary that runs extra init code
> > on the first __getitem__ call. However, the performance of __getitem__
> > is quite important - so I'm trying in the subclassed __getitem__
> > method to first run some code and then patch in the original dict
> > method for the instance to avoid even the check to see if the init
> > code has been run. Various recipes using instancemethod and the like
> > have failed me.
>
> > Curiously if __slots__ is not specified no error occurs when setting
> > self.__getitem__ but the function is not overriden. If __slots__ is
> > ['__getitem__'] however it complains that __getitem__ is read only. I
> > do not understand that behavior.
>
> > --
> > Zachary Burns
> > (407)590-4814
> > Aim - Zac256FL
> > Production Engineer (Digital Overlord)
> > Zindagi Games
>
> That sounds like the State Pattern, from GoF. *http://en.wikipedia.org/wiki/State_pattern
>
> I like the idea of 'renaming', not redefining, but reassigning methods
> at different points during an object's lifetime. *I often wish I had
> more experience with it, and more docs talked about it.
>
> It's hard on memory usage, since each instance has its own function
> attribute, even if there's still only one instance of the function.
> Without it, the function attribute is just looked up on the class.
>
> Not thoroughly tested:
>
> >>> class A:
>
> ... * * def methA( self ):
> ... * * * * * * print 'methA'
> ... * * * * * * self.meth= self.methB
> ... * * meth= methA
> ... * * def methB( self ):
> ... * * * * * * print 'methB'
> ...>>> a= A()
> >>> a.meth()
> methA
> >>> a.meth()
>
> methB

The problem with using this this pattern in the way that you've
specified is that you have a potential memory leak/object lifetime
issue. Assigning a bound method of an instance (which itself holds a
reference to self) to another attribute in that same instance creates
a kind of circular dependency that I have discovered can trip up the
GC more often than not.

You can subclass it as easily:

class dictsubclass(dict):
def __getitem__(self, keyname):
if not hasattr(self, '_run_once'):
self.special_code_to_run_once()
self._run_once = True
return super(self, dict).__getitem__(keyname)

If that extra ~16 bytes associated with the subclass is really a
problem:

class dictsubclass(dict):
def __getitem__(self, keyname):
self.special_code_to_run_once()
self.__class__ = dict
return super(self, dict).__getitem__(keyname)

But I don't think that's a good idea at all.

Re: Overriding a method at the instance level on a subclass of abuiltin type
 

On Dec 3, 1:25*pm, Jason Scheirer <jason.schei...@gmail.com> wrote:
> On Dec 2, 6:13*pm, Aaron Brady <castiro...@gmail.com> wrote:
> > >>> class A:
>
> > ... * * def methA( self ):
> > ... * * * * * * print 'methA'
> > ... * * * * * * self.meth= self.methB
> > ... * * meth= methA
> > ... * * def methB( self ):
> > ... * * * * * * print 'methB'
> > ...>>> a= A()
> > >>> a.meth()
> > methA
> > >>> a.meth()
>
> > methB
>
> The problem with using this this pattern in the way that you've
> specified is that you have a potential memory leak/object lifetime
> issue. Assigning a bound method of an instance (which itself holds a
> reference to self) to another attribute in that same instance creates
> a kind of circular dependency that I have discovered can trip up the
> GC more often than not.
>
> You can subclass it as easily:
>
> class dictsubclass(dict):
> * * def __getitem__(self, keyname):
> * * * * if not hasattr(self, '_run_once'):
> * * * * * * self.special_code_to_run_once()
> * * * * * * self._run_once = True
> * * * * return super(self, dict).__getitem__(keyname)
>
> If that extra ~16 bytes associated with the subclass is really a
> problem:
>
> class dictsubclass(dict):
> * * def __getitem__(self, keyname):
> * * * * self.special_code_to_run_once()
> * * * * self.__class__ = dict
> * * * * return super(self, dict).__getitem__(keyname)
>
> But I don't think that's a good idea at all.

Interesting. The following code ran, and process memory usage rose to
150MB. It failed to return to normal afterward.

>>> for x in range( 10000000 ):
.... a= []
.... a.append(a)
....

However, the following code succeeded in returning usage to normal.

>>> import gc
>>> gc.collect()

It was in version 2.6. So, the GC succeeded in collecting circularly
linked garbage when invoked manually. That might have implications in
the OP's use case.

In another language, it would work differently, if it lacked unbound
method descriptors. C++ for example, untested:

class C {
public:
func_t meth;
C( ) { meth= methA; }
void methA( ) { meth= methB; }
void methB( ) { }
};

It has no problems with memory consumption (an extra pointer per
object), or circular references; functions are not first-class
objects. However they are in Python, which creates an entire bound
method object per instance.

The OP stated:

> run some code and then patch in the original dict
> method for the instance to avoid even the check to see if the init
> code has been run.

So your, Arnaud's, and Bryan's '.__class__' solution is probably best,
and possibly even truer to the intent of the State Pattern.

It is too bad that you can't assign an unbound method to the member,
and derive the bound method on the fly. That might provide a middle-
ground solution.

Re: Overriding a method at the instance level on a subclass of abuiltin type
 

The class method seems to be the most promising, however I have more
'state' methods to worry about so I might end up building new classes
on the fly rather than have a class per permutation of states! Now the
code isn't quite as clear as I thought it was going to be.

It seems unfortunate to me that methods are always looked up on the
class for new style objects. Was this done for speed reasons?
--
Zachary Burns
(407)590-4814
Aim - Zac256FL
Production Engineer (Digital Overlord)
Zindagi Games



On Thu, Dec 4, 2008 at 2:38 AM, Aaron Brady <castironpi@gmail.com> wrote:
> On Dec 3, 1:25 pm, Jason Scheirer <jason.schei...@gmail.com> wrote:
>> On Dec 2, 6:13 pm, Aaron Brady <castiro...@gmail.com> wrote:
>> > >>> class A:
>>
>> > ... def methA( self ):
>> > ... print 'methA'
>> > ... self.meth= self.methB
>> > ... meth= methA
>> > ... def methB( self ):
>> > ... print 'methB'
>> > ...>>> a= A()
>> > >>> a.meth()
>> > methA
>> > >>> a.meth()
>>
>> > methB
>>
>> The problem with using this this pattern in the way that you've
>> specified is that you have a potential memory leak/object lifetime
>> issue. Assigning a bound method of an instance (which itself holds a
>> reference to self) to another attribute in that same instance creates
>> a kind of circular dependency that I have discovered can trip up the
>> GC more often than not.
>>
>> You can subclass it as easily:
>>
>> class dictsubclass(dict):
>> def __getitem__(self, keyname):
>> if not hasattr(self, '_run_once'):
>> self.special_code_to_run_once()
>> self._run_once = True
>> return super(self, dict).__getitem__(keyname)
>>
>> If that extra ~16 bytes associated with the subclass is really a
>> problem:
>>
>> class dictsubclass(dict):
>> def __getitem__(self, keyname):
>> self.special_code_to_run_once()
>> self.__class__ = dict
>> return super(self, dict).__getitem__(keyname)
>>
>> But I don't think that's a good idea at all.
>
> Interesting. The following code ran, and process memory usage rose to
> 150MB. It failed to return to normal afterward.
>
>>>> for x in range( 10000000 ):
> ... a= []
> ... a.append(a)
> ...
>
> However, the following code succeeded in returning usage to normal.
>
>>>> import gc
>>>> gc.collect()
>
> It was in version 2.6. So, the GC succeeded in collecting circularly
> linked garbage when invoked manually. That might have implications in
> the OP's use case.
>
> In another language, it would work differently, if it lacked unbound
> method descriptors. C++ for example, untested:
>
> class C {
> public:
> func_t meth;
> C( ) { meth= methA; }
> void methA( ) { meth= methB; }
> void methB( ) { }
> };
>
> It has no problems with memory consumption (an extra pointer per
> object), or circular references; functions are not first-class
> objects. However they are in Python, which creates an entire bound
> method object per instance.
>
> The OP stated:
>
>> run some code and then patch in the original dict
>> method for the instance to avoid even the check to see if the init
>> code has been run.
>
> So your, Arnaud's, and Bryan's '.__class__' solution is probably best,
> and possibly even truer to the intent of the State Pattern.
>
> It is too bad that you can't assign an unbound method to the member,
> and derive the bound method on the fly. That might provide a middle-
> ground solution.
>
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>

Re: Overriding a method at the instance level on a subclass of a builtin type
 

"Zac Burns" <zac256@gmail.com> writes:

> The class method seems to be the most promising, however I have more
> 'state' methods to worry about so I might end up building new classes
> on the fly rather than have a class per permutation of states! Now the
> code isn't quite as clear as I thought it was going to be.
>
> It seems unfortunate to me that methods are always looked up on the
> class for new style objects. Was this done for speed reasons?

It's only special methods such as __getitem__, ...

You can override normal method on a per-object basis just by adding a
callable attribute with its name to the object:

>>> class A(object):
.... def foo(self): print 'A.foo'
....
>>> a = A()
>>> a.foo()
A.foo
>>> def a_foo(): print 'a.foo'
....
>>> a.foo = a_foo
>>> a.foo()
a.foo

--
Arnaud

Re: Overriding a method at the instance level on a subclass of abuiltin type
 

On Dec 4, 12:31*pm, Arnaud Delobelle <arno...@googlemail.com> wrote:
> "Zac Burns" <zac...@gmail.com> writes:
> > The class method seems to be the most promising, however I have more
> > 'state' methods to worry about so I might end up building new classes
> > on the fly rather than have a class per permutation of states! Now the
> > code isn't quite as clear as I thought it was going to be.
>
> > It seems unfortunate to me that methods are always looked up on the
> > class for new style objects. Was this done for speed reasons?
>
> It's only special methods such as __getitem__, ...
>
> You can override normal method on a per-object basis just by adding a
> callable attribute with its name to the object:
>
> >>> class A(object):
>
> ... * * def foo(self): print 'A.foo'
> ...>>> a = A()
> >>> a.foo()
> A.foo
> >>> def a_foo(): print 'a.foo'
> ...
> >>> a.foo = a_foo
> >>> a.foo()

Note that the overriden "method" here is a plain function; it doesn't
take self as the first argument. If you want to bind it to a callable
that expects the first argument to be self, you have to bind
explicitly self to the object:

>>> def a_foo(self): print 'a.foo'
>>> a.foo = a_foo
>>> a.foo()
TypeError: a_foo() takes exactly 1 argument (0 given)
>>> from functools import partial
>>> a.foo = partial(a_foo,a)
>>> a.foo()
a_foo

George