Dynamically determine base classes on instantiation
 

Hi list,

I'm confronted with a strang problem I cannot find a clean solution
for. To me it seems like I need meta-classes. Anyway, I stucked a bit
deeper in that topic and couldn't find a proper solution neither. But,
judge for yourselve.

I want a class that determines on instantiating its base classes
dynamically. Consider the following two use cases

a = Foo(['a', 'list']) # returns an instance that behaves like a list
assert len(a) == 2
assert a[0] == 'a'
assert a == ['a', 'list']
assert isinstance(a, list) # This would be nice, but no must-have

b = Foo({'blah': 8}) # returns an instance that behaves like a dict
assert b['blah'] == 'blah'
assert b == {'blah': 8}
assert isinstance(b, dict) # again, no must-have

a.do_something() # common function to both instances as defined
b.do_something() # in the Foo class


What I'm currently doing something like the following:

class Foo(object):

def __init__(self, obj):
self._obj = obj

def __len__(self):
return len(self._obj)

def __getitem__(self, name):
return self._obj[name]

# …

def do_something(self):
# do something on self._obj
pass

Which seems ugly. Is there a way to provide the functions of `list'
and `dict' in Foo's look-up path without having to write all the
stubs myself?

Regards,
Thomas Bach.

Re: Dynamically determine base classes on instantiation
 

On Wed, 15 Aug 2012 23:17:41 +0200, Thomas Bach wrote:

> Hi list,
>
> I'm confronted with a strang problem I cannot find a clean solution for.
> I want a class that determines on instantiating its base classes
> dynamically. Consider the following two use cases

Some comments:

1) What you show are not "use cases", but "examples". A use-case is a
description of an actual real-world problem that needs to be solved. A
couple of asserts is not a use-case.

2) You stated that you have a "strange problem", but you haven't told us
what that problem is, you went directly to what you think is the
solution: "a class that determines on instantiating its base classes
dynamically".

How about you tell us the problem, and we'll suggest a solution?

I'm pretty sure it isn't going to be what you asked for, because that
goes completely against the fundamentals of object-oriented design.

Consider your two examples:

a = Foo(['a', 'list'])
b = Foo({'blah': 8})

According to your design:

a is a Foo
b is a Foo
therefore a and b are the same type

So far so good: this is perfectly normal object-oriented design.

But you also have

a is a list, but not a dict
b is a dict, but not a list
therefore a and b are different types

So you contradict yourself: at the same time, a and b are both the same
and different types.

So now you see why you shouldn't do what you ask for. Now let me tell you
why you *can't* do what you ask for: Python's classes don't work like
that. You can't set the base classes of an instance individually. All
instances of a class share the same base classes.

I think that the right solution here is not inheritance, but composition
and delegation. You're already on the right track when you give your Foo
instances an attribute _obj and then operate on that, but you are wrong
to focus on inheritance.

Instead, Foo should implement only the shared operations, and everything
else should be delegated to _obj.

Automatic delegation is trivially easy (except see below):

http://code.activestate.com/recipes/52295

This is a *really old* recipe, from ancient days before you could inherit
from built-in types like list, dict etc., so the description of the
problem is no longer accurate. But the technique is still good, with
unfortunately one complication:

If you inherit from builtins, you cannot use automatic delegation on the
magic "double-underscore" (dunder) methods like __eq__, __len__, etc.

See this thread here for one possible solution:

http://www.velocityreviews.com/forum...ython-3-a.html



--
Steven

Re: Dynamically determine base classes on instantiation
 

On Thu, Aug 16, 2012 at 12:16:03AM +0000, Steven D'Aprano wrote:
> Some comments:
>
> 1) What you show are not "use cases", but "examples". A use-case is a
> description of an actual real-world problem that needs to be solved. A
> couple of asserts is not a use-case.

Thanks for the clarification on that one. So, here's the use-case: I'm
querying the crunchbase API which returns JSON data and is rather
poorly documented. I want to create a data model for the companies
listed on Crunchbase in order to be able to put the queried data in a
data-base. As I am too lazy to examine all the data by hand I thought
I automatize this. I thought that it would be nice to be able to pass
a function a parsed JSON object (AFAIK these are lists, dicts,
strings, ints, floats, strs in Python) and it returns me the type of
these objects. For the simple classes (str, int, float) this is quite
trivial: F('foo') should return `str' and F(8) should return `int'.

For a compound object like dict I would like it to return the data
fields with their type. Hence, F({'foo': 8}) should return
{'foo': int}, and given that f = F({'foo': {'bar': 80}}) I would like
f to equal to {'foo': dict}, with the option to query the type of
'foo' via f.foo, where the latter should equal to {'bar': int}. So
far, this is not a complicated case. But, sometimes a data field on
returned data set is simply None. Thus, I want to extract the types from
another data set and merge the two.

So, my question (as far as I can see it, please correct me if I am
wrong) is less of the "How do I achieve this?"-kind, but more of the
"What is a clean design for this?"-kind. My intuitive thought was that
the `merge' function should be a part of the object returned from `F'.

> How about you tell us the problem, and we'll suggest a solution?

I can see your point. On the other hand, by expressing my thoughts you
can at least tell me that these are completely wrong and correct my
way of thinking this way.

> Consider your two examples:
>
> a = Foo(['a', 'list'])
> b = Foo({'blah': 8})
>
> According to your design:
>
> a is a Foo
> b is a Foo

I actually never said that. I simply wanted `a' and `b' to share the
same function (the `merge' function), I thought that the easiest way
to achieve this is by letting them share the same name-space. But, as
you show: …

> therefore a and b are the same type
>
> So far so good: this is perfectly normal object-oriented design.
>
> But you also have
>
> a is a list, but not a dict
> b is a dict, but not a listn
> therefore a and b are different types
>
> So you contradict yourself: at the same time, a and b are both the same
> and different types.

… I already made a mistake on the logical level.

> Instead, Foo should implement only the shared operations, and everything
> else should be delegated to _obj.
>
> If you inherit from builtins, you cannot use automatic delegation on the
> magic "double-underscore" (dunder) methods like __eq__, __len__, etc.
>
> See this thread here for one possible solution:
>
> http://www.velocityreviews.com/forum...ython-3-a.html
>

OK, thanks for the hint. I will see how I'm going to put all this
stuff together.

Regards,
Thomas.

Re: Dynamically determine base classes on instantiation
 

On 16/08/12 14:52:30, Thomas Bach wrote:
> On Thu, Aug 16, 2012 at 12:16:03AM +0000, Steven D'Aprano wrote:
>> > Some comments:
>> >
>> > 1) What you show are not "use cases", but "examples". A use-case is a
>> > description of an actual real-world problem that needs to be solved. A
>> > couple of asserts is not a use-case.
> Thanks for the clarification on that one. So, here's the use-case: I'm
> querying the crunchbase API which returns JSON data and is rather
> poorly documented. I want to create a data model for the companies
> listed on Crunchbase in order to be able to put the queried data in a
> data-base. As I am too lazy to examine all the data by hand I thought
> I automatize this. I thought that it would be nice to be able to pass
> a function a parsed JSON object (AFAIK these are lists, dicts,
> strings, ints, floats, strs in Python) and it returns me the type of
> these objects. For the simple classes (str, int, float) this is quite
> trivial: F('foo') should return `str' and F(8) should return `int'.
>
> For a compound object like dict I would like it to return the data
> fields with their type. Hence, F({'foo': 8}) should return
> {'foo': int}, and given that f = F({'foo': {'bar': 80}}) I would like
> f to equal to {'foo': dict}, with the option to query the type of
> 'foo' via f.foo, where the latter should equal to {'bar': int}. So
> far, this is not a complicated case. But, sometimes a data field on
> returned data set is simply None. Thus, I want to extract the types from
> another data set and merge the two.
>
> So, my question (as far as I can see it, please correct me if I am
> wrong) is less of the "How do I achieve this?"-kind, but more of the
> "What is a clean design for this?"-kind. My intuitive thought was that
> the `merge' function should be a part of the object returned from `F'.

The misunderstanding is that you feel F should return an object with
a 'merge' method and a varying abse type, while Steven and others
think that F should be a function.

Maybe something like:

def F(obj):
if obj is None:
return None
tp = type(obj)
if tp in (bool, int, float, str):
return tp
elif tp is list:
return merge([F(elem) for elem in obj])
elif tp is dict:
return dict((k, F(v)) for k,v in obj.iteritems())
else:
raise ValueError("Unexpected type %s for value %s" %(tp, obj))

def merge(lst):
if None in lst:
not_nones = [elem for elem in lst if elem is not None]
if not_nones:
not_none = not_nones[0]
lst = [not_none if elem is None else elem for elem in lst]
else:
return lst # all elements are None; nothing can be done
types = {}
for elem in lst:
if type(elem) is dict:
for k,v in elem.iteritems():
if v is None:
if k in types:
elem[k] = types[k]
else:
for other in lst:
if (other is not elem
and type(other) is dict
and k in other
and other[k] is not None
):
elem[k] = types[k] = other[k]
break
return lst


The merge logic you have in mind may be different from what I just
made up, but the idea remains: F and merge can be functions.


Hope this helps,

-- HansM

Re: Dynamically determine base classes on instantiation
 

On Thu, 16 Aug 2012 14:52:30 +0200, Thomas Bach
<thbach@students.uni-mainz.de> declaimed the following in
gmane.comp.python.general:

> Thanks for the clarification on that one. So, here's the use-case: I'm
> querying the crunchbase API which returns JSON data and is rather
> poorly documented. I want to create a data model for the companies
> listed on Crunchbase in order to be able to put the queried data in a
> data-base. As I am too lazy to examine all the data by hand I thought
> I automatize this. I thought that it would be nice to be able to pass
> a function a parsed JSON object (AFAIK these are lists, dicts,
> strings, ints, floats, strs in Python) and it returns me the type of
> these objects. For the simple classes (str, int, float) this is quite
> trivial: F('foo') should return `str' and F(8) should return `int'.
>
I'm not familiar with JSON structure, but off-hand I'd say the point
to determine the nature of the data is during the so-called parsing of
the JSON data itself, not after... Based upon http://www.json.org/ the
type of an item can basically be determined from the first character of
the "value":
{ dictionary (JSON "object")
[ list (JSON "array")
" string
t/f/n true/false/null
- or 0..9 number

I'd be looking for some way to have the parser itself return a
structure of tuples of (type, parsedJSONitem)

Of course, since the parse result (at least from my recent
experiment) is a Python structure, it isn't difficult to walk that
structure...

>>> import simplejson as j
>>> SAMPLE = '["foo", {"bar":["baz", null, 1.0, 2]}]'
>>> parse = j.loads(SAMPLE)
>>> parse
[u'foo', {u'bar': [u'baz', None, 1.0, 2]}]
>>> def typer(obj, level=0):
.... otype = type(obj)
.... print "\t" * level, otype
.... if otype == type([]):
.... for o in obj:
.... typer(o, level+1)
.... elif otype == type({}):
.... for k,o in obj.items():
.... print "\t" * level, " ", k
.... typer(o, level+1)
.... else:
.... print "\t" * level, " ", obj
....
>>> print parse
[u'foo', {u'bar': [u'baz', None, 1.0, 2]}]
>>> typer(parse)
<type 'list'>
<type 'unicode'>
foo
<type 'dict'>
bar
<type 'list'>
<type 'unicode'>
baz
<type 'NoneType'>
None
<type 'float'>
1.0
<type 'int'>
2
>>>

{Hmmm, forgot to print the type of "k" for dictionaries}.

> For a compound object like dict I would like it to return the data
> fields with their type. Hence, F({'foo': 8}) should return
> {'foo': int}, and given that f = F({'foo': {'bar': 80}}) I would like
> f to equal to {'foo': dict}, with the option to query the type of
> 'foo' via f.foo, where the latter should equal to {'bar': int}. So
> far, this is not a complicated case. But, sometimes a data field on
> returned data set is simply None. Thus, I want to extract the types from
> another data set and merge the two.
>

"But, sometimes a data field on returned data set is simply None.
Thus, I want to extract the types from another data set and merge the
two." ??? A "data field" /value/ of None has the /type/ "<type
'NoneType'>", so I don't quite understand what you intend to merge? You
can't arbitrarily change the "type" without changing the "value".
--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com HTTP://wlfraed.home.netcom.com/

Re: Dynamically determine base classes on instantiation
 

On Thu, Aug 16, 2012 at 05:10:43PM +0200, Hans Mulder wrote:
> On 16/08/12 14:52:30, Thomas Bach wrote:
> >
> > So, my question (as far as I can see it, please correct me if I am
> > wrong) is less of the "How do I achieve this?"-kind, but more of the
> > "What is a clean design for this?"-kind. My intuitive thought was that
> > the `merge' function should be a part of the object returned from `F'.
>
> The misunderstanding is that you feel F should return an object with
> a 'merge' method and a varying abse type, while Steven and others
> think that F should be a function.

OK, then my design wasn't so bad in the first place. :)

I made a class `Model' which wraps the actual type and realized
`merge' and `F' (with a better name, though) as classmethods of
`Model' in order to tie together the stuff that belongs together. By
the way, another need I saw for this design was that

setattr(Model(), 'foo', {'bar': int})

works, whereas

setattr(dict(), 'foo', {'bar': int})

raises an AttributeError (on Python 3.2). Could someone give me the
buzz word (or even an explanation) on why that is so?

Thomas Bach

Re: Dynamically determine base classes on instantiation
 

class Foo(object):
def __new__(cls, arg):
if isinstance(arg, list):
cls = FooList
elif isinstance(arg, dict):
cls = FooDict
return object.__new__(cls, arg)

class FooList(Foo, list):
pass

class FooDict(Foo, dict):
pass

You could even have __new__ make these Foo* classes dynamically when it encounters a new type of argument.

Chard.

On Thursday, 16 August 2012 18:54:12 UTC+2, Thomas Bach wrote:
> On Thu, Aug 16, 2012 at 05:10:43PM +0200, Hans Mulder wrote:
>
> > On 16/08/12 14:52:30, Thomas Bach wrote:
>
> > >
>
> > > So, my question (as far as I can see it, please correct me if I am
>
> > > wrong) is less of the "How do I achieve this?"-kind, but more of the
>
> > > "What is a clean design for this?"-kind. My intuitive thought was that
>
> > > the `merge' function should be a part of the object returned from `F'.
>
> >
>
> > The misunderstanding is that you feel F should return an object with
>
> > a 'merge' method and a varying abse type, while Steven and others
>
> > think that F should be a function.
>
>
>
> OK, then my design wasn't so bad in the first place. :)
>
>
>
> I made a class `Model' which wraps the actual type and realized
>
> `merge' and `F' (with a better name, though) as classmethods of
>
> `Model' in order to tie together the stuff that belongs together. By
>
> the way, another need I saw for this design was that
>
>
>
> setattr(Model(), 'foo', {'bar': int})
>
>
>
> works, whereas
>
>
>
> setattr(dict(), 'foo', {'bar': int})
>
>
>
> raises an AttributeError (on Python 3.2). Could someone give me the
>
> buzz word (or even an explanation) on why that is so?
>
>
>
> Thomas Bach

Re: Dynamically determine base classes on instantiation
 

class Foo(object):
def __new__(cls, arg):
if isinstance(arg, list):
cls = FooList
elif isinstance(arg, dict):
cls = FooDict
return object.__new__(cls, arg)

class FooList(Foo, list):
pass

class FooDict(Foo, dict):
pass

You could even have __new__ make these Foo* classes dynamically when it encounters a new type of argument.

Chard.

On Thursday, 16 August 2012 18:54:12 UTC+2, Thomas Bach wrote:
> On Thu, Aug 16, 2012 at 05:10:43PM +0200, Hans Mulder wrote:
>
> > On 16/08/12 14:52:30, Thomas Bach wrote:
>
> > >
>
> > > So, my question (as far as I can see it, please correct me if I am
>
> > > wrong) is less of the "How do I achieve this?"-kind, but more of the
>
> > > "What is a clean design for this?"-kind. My intuitive thought was that
>
> > > the `merge' function should be a part of the object returned from `F'.
>
> >
>
> > The misunderstanding is that you feel F should return an object with
>
> > a 'merge' method and a varying abse type, while Steven and others
>
> > think that F should be a function.
>
>
>
> OK, then my design wasn't so bad in the first place. :)
>
>
>
> I made a class `Model' which wraps the actual type and realized
>
> `merge' and `F' (with a better name, though) as classmethods of
>
> `Model' in order to tie together the stuff that belongs together. By
>
> the way, another need I saw for this design was that
>
>
>
> setattr(Model(), 'foo', {'bar': int})
>
>
>
> works, whereas
>
>
>
> setattr(dict(), 'foo', {'bar': int})
>
>
>
> raises an AttributeError (on Python 3.2). Could someone give me the
>
> buzz word (or even an explanation) on why that is so?
>
>
>
> Thomas Bach

Re: Dynamically determine base classes on instantiation
 

> a is a Foo
> b is a Foo
> therefore a and b are the same type

What you mean here is "a and b share a common base class".

Re: Dynamically determine base classes on instantiation
 

On Thu, Aug 16, 2012 at 12:29:21PM -0400, Dennis Lee Bieber wrote:
> On Thu, 16 Aug 2012 14:52:30 +0200, Thomas Bach
> <thbach@students.uni-mainz.de> declaimed the following in
> gmane.comp.python.general:
>
> Of course, since the parse result (at least from my recent
> experiment) is a Python structure, it isn't difficult to walk that
> structure...

I prefer that one, as I have the parsed data already lying around in
memory. But, as I think about it, I could also pass it to json.dumps
and parse it again. But, that wouldn't make much sense, right?

>
> "But, sometimes a data field on returned data set is simply None.
> Thus, I want to extract the types from another data set and merge the
> two." ??? A "data field" /value/ of None has the /type/ "<type
> 'NoneType'>", so I don't quite understand what you intend to merge? You
> can't arbitrarily change the "type" without changing the "value".

OK, I am probably using the wrong vocabulary here again. :(

Imagine you have two data sets:

d1 = {'foo': None}
d2 = {'foo': 8}

Where I would assume that d1 has "foo" not set. That's why I want this
whole "merge"-thing in the first place: to be able to extract the type
{'foo': None} from d1 and {'foo': int} from d2 and merge the two
together which should result in {'foo': int}.

Regards,
Thomas Bach.