Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   subclassing extension type and assignment to __class__ (http://www.velocityreviews.com/forums/t338506-subclassing-extension-type-and-assignment-to-__class__.html)

gregory lielens 11-27-2004 01:19 PM

subclassing extension type and assignment to __class__
 
Hello,

I am currently writing python bindings to an existing C++ library, and I
just encountered a problem that I hope has been solved by more
experienced python programmers:

A C++ class (let's call it CClass) is binded using classical Python
extension API to _PClass, which is accesible through python without any
problem. The problem is that I want this class to be extended/extensible
in python, and expose the python-extended version (PClass) to library
users (_PClass should never be used directly nor be retruned by any
function).
The aim is to leave only performance critical methods in C++ so that the
binding work is minimal, and develop the other methods in python so that
they are easier to maintain/extend.

We thus have something like this

class PClass(_PClass):
def overide_method(self,...):
...
def new_method(self,...):
...

and I can define
a=PClass()
and use my new or overiden method
a.overide_method() a.new_method() as intended...

So far, so good, trouble begin when I have a method from another class
PClass2 derived from _PClass2 which bind the C++ class CClass2, that
should return objects of type PClass:

Let call this method troublesome_method:

b=_PClass2()
c=b.troublesome_method()
type(c) gives _PClass.

now I want to define a python class PClass2 for extending methods of
_PClass2 like I have done for _PClass, in particular I want that PClass2
troublesome_method return objects of type PClass instead of _PClass...

To this end I try something like this

class PClass2(_PClass2):
...
def troubelsome_method(self):
base=_PClass2.troublesome_method(self)
base.__class__=PClass

and I have python complaining about TypeError: __class__ assignment:
only for heap types...

It seems thus that my approach is not possible, but problem is that I
can not see another approach that would not involve far more efforts
that this one (I have the impression I have to forget about making my
PClass derived from _PClass, but instead embed a _PClass instance in
PClass, far from ideal cause I would have to reimplement all method, not
only those that I want to change :( )
This is particularly frustrating cause I also have the impression that
want I want to do was at one time possible in python, let say in
2002-2003, when __class__ was already assignable but before assignment
was forbidden for non-heap types

Any hint on this problem?

Thanks,

Greg.

Lenard Lindstrom 11-27-2004 08:22 PM

Re: subclassing extension type and assignment to __class__
 
gregory lielens <gregory.lielens@fft.be> writes:

> Hello,
>
> I am currently writing python bindings to an existing C++ library, and
> I just encountered a problem that I hope has been solved by more
> experienced python programmers:
>
> A C++ class (let's call it CClass) is binded using classical Python
> extension API to _PClass, which is accesible through python without
> any problem. The problem is that I want this class to be
> extended/extensible in python, and expose the python-extended version
> (PClass) to library users (_PClass should never be used directly nor
> be retruned by any function).
> The aim is to leave only performance critical methods in C++ so that
> the binding work is minimal, and develop the other methods in python
> so that they are easier to maintain/extend.
>
> We thus have something like this
>
> class PClass(_PClass):
> def overide_method(self,...):
> ...
> def new_method(self,...):
> ...
>
> and I can define
> a=PClass()
> and use my new or overiden method
> a.overide_method() a.new_method() as intended...
>
> So far, so good, trouble begin when I have a method from another class
> PClass2 derived from _PClass2 which bind the C++ class CClass2, that
> should return objects of type PClass:
>
> Let call this method troublesome_method:
>
> b=_PClass2()
> c=b.troublesome_method()
> type(c) gives _PClass.
>
> now I want to define a python class PClass2 for extending methods of
> _PClass2 like I have done for _PClass, in particular I want that
> PClass2 troublesome_method return objects of type PClass instead of
> _PClass...
>
> To this end I try something like this
>
> class PClass2(_PClass2):
> ...
> def troubelsome_method(self):
> base=_PClass2.troublesome_method(self)
> base.__class__=PClass
>
> and I have python complaining about TypeError: __class__ assignment:
> only for heap types...
>
> It seems thus that my approach is not possible, but problem is that I
> can not see another approach that would not involve far more efforts
> that this one (I have the impression I have to forget about making my
> PClass derived from _PClass, but instead embed a _PClass instance in
> PClass, far from ideal cause I would have to reimplement all method,
> not only those that I want to change :( )
> This is particularly frustrating cause I also have the impression that
> want I want to do was at one time possible in python, let say in
> 2002-2003, when __class__ was already assignable but before assignment
> was forbidden for non-heap types
>
> Any hint on this problem?
>

I have just run into the same thing with the builtin list type. The trick
was not to declare my subclass directly from list, but rather a subclass
of list. So try this:

class PClassBase(_PClass):
pass

class PClass(PClassBase):
...

class PClass2(PClassBase):
...

Why? I cannot say other than I have noted that Python new-style classes
and extension types are not quite the same thing. That is, a new-style
class is a particular kind of extension type.

Lenard Lindstrom
<len-l@telus.net>

gregory lielens 11-27-2004 11:50 PM

Re: subclassing extension type and assignment to __class__
 
Thanks for your answer, it means I am not the only one having this kind
of problem...
> I have just run into the same thing with the builtin list type. The trick
> was not to declare my subclass directly from list, but rather a subclass
> of list. So try this:
>
> class PClassBase(_PClass):
> pass
>
> class PClass(PClassBase):
> ...
>
> class PClass2(PClassBase):
> ...
>

I think here you mean PClass2(PClassBase2), with a
PClassBase2(_PClass2): In my example the 2 class do not inherit fro the
same base class...
> Why? I cannot say other than I have noted that Python new-style classes
> and extension types are not quite the same thing. That is, a new-style
> class is a particular kind of extension type.


I see how it can help me making PClassBase a PClass (or any type derived
from PClassBase), but what I fail to see is how this will allow me to
change a _PClass into one of it's derived types: this is necessary for
updating methods of _PClass2 that return object of type _PClass (and on
this I do not have much control, both _PClass and _PClass2 and all of
their methods are implemented in C++). To updates theses methods for the
pure python derived class PClass2, I have to make them return the python
class PClass instead of the original _PClass, which means I still have
to transform a _PClass object into its derived type PClassBase or
PClass, and at this time things go pear-shaped :-(....or is there
something I miss?



Greg.

Lenard Lindstrom 11-28-2004 07:02 PM

Re: subclassing extension type and assignment to __class__
 
gregory lielens <gregory.lielens@fft.be> writes:

> Thanks for your answer, it means I am not the only one having this
> kind of problem...
> > I have just run into the same thing with the builtin list type. The trick
> > was not to declare my subclass directly from list, but rather a subclass
> > of list. So try this:
> > class PClassBase(_PClass):
> > pass
> > class PClass(PClassBase):
> > ...
> > class PClass2(PClassBase):
> > ...
> >

> I think here you mean PClass2(PClassBase2), with a
> PClassBase2(_PClass2): In my example the 2 class do not inherit fro
> the same base class...
> > Why? I cannot say other than I have noted that Python new-style classes
> > and extension types are not quite the same thing. That is, a new-style
> > class is a particular kind of extension type.

>
> I see how it can help me making PClassBase a PClass (or any type
> derived from PClassBase), but what I fail to see is how this will
> allow me to change a _PClass into one of it's derived types: this is
> necessary for updating methods of _PClass2 that return object of type
> _PClass (and on this I do not have much control, both _PClass and
> _PClass2 and all of their methods are implemented in C++). To updates
> theses methods for the pure python derived class PClass2, I have to
> make them return the python class PClass instead of the original
> _PClass, which means I still have to transform a _PClass object into
> its derived type PClassBase or PClass, and at this time things go
> pear-shaped :-(....or is there something I miss?
>

Sorry for the confusion. This looks like a situation best handled with
embedding rather than inheritance. The PClass and PClass2 wrapper classes
can share whatever relationship _PClass and _PClass2 share.

class PClass:
def __init__(inst=None):
if inst is None:
inst = _PClass()
self._inst = inst
def override_method(self, ...):
self._inst.override_method(...)
def new_method(self, ...):
...

class PClass2:
self._inst = _PClass2()
def troublesome_method(self):
base = PClass(self._inst.troublesome_method())
...
return base

Alternatively you could provide a way to pass PClass back to the extension module
for _PClass2.troublesome_method to use instead of _PClass when creating a return
object. But the embedding approach seems more appropriate when _PClass is not to
be used directly.

Lenard Lindstrom
<len-l@telus.net>

gregory lielens 11-28-2004 09:45 PM

Re: subclassing extension type and assignment to __class__
 

> Sorry for the confusion. This looks like a situation best handled with
> embedding rather than inheritance. The PClass and PClass2 wrapper classes
> can share whatever relationship _PClass and _PClass2 share.
>
> class PClass:
> def __init__(inst=None):
> if inst is None:
> inst = _PClass()
> self._inst = inst
> def override_method(self, ...):
> self._inst.override_method(...)
> def new_method(self, ...):
> ...
>
> class PClass2:
> self._inst = _PClass2()
> def troublesome_method(self):
> base = PClass(self._inst.troublesome_method())
> ...
> return base


Yes I was affraid this would be the conclusion: embedding instead of
inheritance...But this means that not only the method that need to be
modified need to be rewritten in python, all the other ones also just to
delegate to the embedded instance...
This is not too practical, even if such writing can be more or less
automatized...
For my purpose it seems that there was a big step backward between
python 2.2 and 2.3, and I am surprised this has been mostly
unnoticed...This problem should be common for many people trying to
write mixed C/C++ / python classes. I found some reference on this in
the list archive, a few people beeing anoyed by it, but the proposition
to make the test about assignment to class less strict was not retained
it seems...I plan to experiment a little with the python C
implementation to see what happen if I bypass this test...

> Alternatively you could provide a way to pass PClass back to the extension module
> for _PClass2.troublesome_method to use instead of _PClass when creating a return
> object. But the embedding approach seems more appropriate when _PClass is not to
> be used directly.


Yes but again this is not so practical, the purpose of the _PClass /
PClass implementation was to make the bindings as light, and as easy to
implement / extend as possible, and both the embedding and the creation
of PClass within _PClass2 methods defeat this purpose...

I think allowing assignment to class for native types should be done if
possible, for example if the new class is a derived type (possibly using
__slots__)... I saw a post on this by Guido, but the reason why it was
not possible is not clear to me and the thread died without further
explanations...Maybe my experiments will show me what happen, or someone
more knowledgeable than me can explain why this is not possible?

Greg.

David Bolen 11-30-2004 12:23 AM

Re: subclassing extension type and assignment to __class__
 
gregory lielens <gregory.lielens@fft.be> writes:

(...)
> Yes I was affraid this would be the conclusion: embedding instead of
> inheritance...But this means that not only the method that need to be
> modified need to be rewritten in python, all the other ones also just
> to delegate to the embedded instance...
> This is not too practical, even if such writing can be more or less
> automatized...


Unless I'm misunderstanding, couldn't one of __getattr__ or
__getattribute__ make mapping other methods you don't override very
simple and practical, not to mention fully automated with 2-3 lines of
code? In particular, __getattr__ would seem good for your use since
it is only called for attributes that couldn't already be located.

I've had code that wrapped underlying objects in a number of cases, and
always found that to be a pretty robust mechanism.

-- David

Gregory Lielens 12-02-2004 10:45 AM

Re: subclassing extension type and assignment to __class__
 

> Unless I'm misunderstanding, couldn't one of __getattr__ or
> __getattribute__ make mapping other methods you don't override very
> simple and practical, not to mention fully automated with 2-3 lines of
> code? In particular, __getattr__ would seem good for your use since
> it is only called for attributes that couldn't already be located.
>
> I've had code that wrapped underlying objects in a number of cases, and
> always found that to be a pretty robust mechanism.


Thanks for mentioning this, after more research I came up with something
usable, using delegating technique from python cookbook:


#wrapper

def wrap_class(base):
class wrapper:
__base__=base
def __init__(self,*args,**kwargs):
if len(args)==1 and type(args[0])==self.__base__ and kwargs=={}:
self._base = args[0]
else:
self._base=self.__class__.__base__.__new__(self.__ class__.__base__,*args,**kwargs)
def __getattr__(self,s):
return self._base.__getattribute__(s)
return wrapper



#wrap class
complex2=wrap_class(complex)
#extend wrapper class
def supaprint(self):
print "printed with supaprint(tm):"
print self
complex2.supaprint=supaprint

#define wrapper class from base class
c1=1+1j
c2=complex2(c1)
#test old functionalities
print c1==c2
print "c1=",c1
print "c2=",c1
#test extension
c2.supaprint()
c1.supaprint()

So this basically fit the bill, it even delegates special methods it
seems, although I am not sure why...It is like writting heritage ourself,
in a way :-)

Remaning problem is that if we use the class generating
wrapper function (simple), we loose the classic class definition syntax
and rely on explicitely adding methods.
The alternative is to include the wrapper
machinery in every "home-derived" class, but you are right, it is not as bad as I
though :-)

The biggest point I am not sure now is performance: Isn't a big penalty
associated to this embedding, compared to derivation? Python performance
is not so critical in our application, but I would be uncomfortable having
a factor 10 penalty in methd calling associated to this approach...For now, this will be used for
testing new implementations/extensions, that will be added to the C++
written binding afterwards.

I'd like to thanks the list for the hints, I will post the results of my
experimatations relaxing the assigment to __class__ test if they are
interesting.

Greg.



All times are GMT. The time now is 08:16 AM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.