«

I'm confused about the following. The idea here is that the set of
instances of some class are small and finite, so I'd like to create them
at class creation time, then hijack __new__ to simply return one of the
preexisting classes instead of creating a new one each call.

This seems to work in python3, but fails in python2 with:

Foo(23)
TypeError: unbound method __Foo__new__() must be called with Foo
instance as first argument (got type instance instead)

I don't understand. Can anyone explain?

--rich

On Mon, 26 Dec 2011 20:28:26 -0800, K. Richard Pixley wrote:

> I'm confused about the following. The idea here is that the set of
> instances of some class are small and finite, so I'd like to create them
> at class creation time, then hijack __new__ to simply return one of the
> preexisting classes instead of creating a new one each call.

I'm surprised it works in Python3. Try this in Python 2 instead:

Foo.__new__ = staticmethod(__Foo__new__)



--
Steven

On 12/26/11 20:53 , Steven D'Aprano wrote:
> On Mon, 26 Dec 2011 20:28:26 -0800, K. Richard Pixley wrote:
>
>> I'm confused about the following. The idea here is that the set of
>> instances of some class are small and finite, so I'd like to create them
>> at class creation time, then hijack __new__ to simply return one of the
>> preexisting classes instead of creating a new one each call.
>
> I'm surprised it works in Python3. Try this in Python 2 instead:
>
> Foo.__new__ = staticmethod(__Foo__new__)

Excellent. Thank you.

--rich

On Mon, 26 Dec 2011, K. Richard Pixley wrote:
> I don't understand. Can anyone explain?

I'm also a bit confused about __new__. I'd very much appreciate it if
someone could explain the following aspects of it:

* The manual (<http://docs.python.org/reference/datamodel.html>) says
that __new__ is "a static method (special-cased so you need not declare
it as such)". What does "special-cased" mean? Apparently, for
instance, in OP's case, Python did not automatically detect that it
should not be bound as a method.

* Is there any part of the manual that explains, holistically, the
greater context of object instantiation into which __new__ fits? I can
only find small parts of it spread around the documentation for __new__
and __init__, but no complete explanation of it. There are several
things I'm wondering about, like what it means to call a type object at
all; how methods, properties and the like are bound; how pickle can
instantiate a class without calling __init__; when and whether __dict__
is created and a couple of other things. Is there a complete
documentation of the process anywhere that I haven't managed to find?

--

Fredrik Tolf

On Mon, Dec 26, 2011 at 10:48 PM, Fredrik Tolf <> wrote:
> I'm also a bit confused about __new__. I'd very much appreciate it if
> someone could explain the following aspects of it:
>
> ** The manual (<http://docs.python.org/reference/datamodel.html>) says
> * that __new__ is "a static method (special-cased so you need not declare
> * it as such)". What does "special-cased" mean? Apparently, for
> * instance, in OP's case, *Python did not automatically detect that it
> * should not be bound as a method.

It apparently has to do with the class creation. For the
special-casing to happen, the __new__ method has to be present when
the class is created. If it is, then it automatically gets wrapped in
a staticmethod. In the OP's case, he was adding the __new__ method
after class creation, so the wrapping did not happen automatically.
Compare:

>>> def my_new(cls): return object.__new__(cls)
....
>>> class Foo(object):
.... __new__ = my_new
....
>>> class Bar(object): pass
....
>>> Bar.__new__ = my_new
>>> Foo.__dict__['__new__']
<staticmethod object at 0x0237D6F0>
>>> Bar.__dict__['__new__']
<function my_new at 0x02381430>

> ** Is there any part of the manual that explains, holistically, the
> * greater context of object instantiation into which __new__ fits? I can
> * only find small parts of it spread around the documentation for __new__
> * and __init__, but no complete explanation of it. There are several
> * things I'm wondering about, like what it means to call a type object at
> * all;

I don't know of anything that organizes it that way specifically, but
I believe the Data Model reference pretty much covers what you're
looking for. From the type hierarchy, under "Callable Types":
http://docs.python.org/reference/dat...type-hierarchy
"""
Class Types
Class types, or “new-style classes,” are callable. These objects
normally act as factories for new instances of themselves, but
variations are possible for class types that override __new__(). The
arguments of the call are passed to __new__() and, in the typical
case, to __init__() to initialize the new instance.
"""

AFAIK, that's pretty much it. When a type is called, __new__ is
called to create the new instance, and then __init__ is called to
initialize it (if __new__ returned an instance of the type).

> how methods, properties and the like are bound;

When they're accessed, using the descriptor protocol, not as part of
the instantiation process. See:
http://docs.python.org/reference/dat...ng-descriptors

> how pickle can
> * instantiate a class without calling __init__;

By calling the __new__ method directly instead of calling the type:
http://docs.python.org/library/pickl...__getnewargs__

> when and whether __dict__
> * is created and a couple of other things.

Under the hood as part of the object creation process, unless the
class uses __slots__.

On 12/27/2011 04:48 PM, Fredrik Tolf wrote:
> On Mon, 26 Dec 2011, K. Richard Pixley wrote:
>> I don't understand. Can anyone explain?
>
> I'm also a bit confused about __new__. I'd very much appreciate it if
> someone could explain the following aspects of it:
>
> * The manual (<http://docs.python.org/reference/datamodel.html>) says
> that __new__ is "a static method (special-cased so you need not declare
> it as such)". What does "special-cased" mean? Apparently, for
> instance, in OP's case, Python did not automatically detect that it
> should not be bound as a method.

If you declare new in the regular way:

class Foo(object):
def __new__(cls):
...

Python would create __new__ as a static method even without applying the
staticmethod decorator; that python does not detect to special case
__new__ when it is added dynamically is probably an oversight in Python
2.x that was fixed in Python 3.x.

On 12/26/11 21:48 , Fredrik Tolf wrote:
> On Mon, 26 Dec 2011, K. Richard Pixley wrote:
>> I don't understand. Can anyone explain?
>
> I'm also a bit confused about __new__. I'd very much appreciate it if
> someone could explain the following aspects of it:
>
> * The manual (<http://docs.python.org/reference/datamodel.html>) says
> that __new__ is "a static method (special-cased so you need not declare
> it as such)". What does "special-cased" mean? Apparently, for
> instance, in OP's case, Python did not automatically detect that it
> should not be bound as a method.
>
> * Is there any part of the manual that explains, holistically, the
> greater context of object instantiation into which __new__ fits? I can
> only find small parts of it spread around the documentation for __new__
> and __init__, but no complete explanation of it. There are several
> things I'm wondering about, like what it means to call a type object at
> all; how methods, properties and the like are bound; how pickle can
> instantiate a class without calling __init__; when and whether __dict__
> is created and a couple of other things. Is there a complete
> documentation of the process anywhere that I haven't managed to find?

Everything I know about this stuff I got by reading the manual. Search
for __init__ and you'll find a pretty complete description of how
objects are created, how to perturb that process, how attributes are
looked up, and how to perturb that process. At least, you do in both
python 2 and python 3 current manuals, I think.

The conceptual leap for me was in recognizing that a class is just an
object. The best way, (imo, so far), to create a singleton in python is
to use the class itself as the singleton rather than ever instantiating
it. With a little work, you can even prevent it from ever being
instantiated.

The concept of a "factory" gets a little weird then, as a factory might
return an instantiation of some class but it might instead return a
class proper - perhaps even an automatically constructed class - which
is a somewhat different sort of factory.

Calling a type is a little bit different. Calling a class is how you
initiate an instantiation of that class. Calling an instantiation leads
to __call__, (which may or may not have much semantic meaning, depending
on your class). It's the difference between C() and C()(). (And the
analogy holds for other builtin types, I think).

Super() is also fairly well documented.

But __metaclass__ could use a few more examples, imo. I'm still not
entirely clear on __metaclass__. As I understand it, (which may well be
completely wrong), __metaclass_ is the means of perturbing the results
of "isinstance". You can create, (or delete), inheritance arbitrarily,
and without even requiring real, existing classes to do it. I'm not
clear on why you'd want to do this, nor why __metaclass__ is a better
mechanism than, say, any of the implementations of "interface" which
have been done in various tool kits or "mixins", which all seem to be
somewhat overlapping in functionality.

--rich

On Tue, 27 Dec 2011, Ian Kelly wrote:
> On Mon, Dec 26, 2011 at 10:48 PM, Fredrik Tolf <> wrote:
>> I'm also a bit confused about __new__. I'd very much appreciate it if
>> someone could explain the following aspects of it:
>>
>> ** The manual (<http://docs.python.org/reference/datamodel.html>) says
>> * that __new__ is "a static method (special-cased so you need not declare
>> * it as such)". What does "special-cased" mean? Apparently, for
>> * instance, in OP's case, *Python did not automatically detect that it
>> * should not be bound as a method.
>
> It apparently has to do with the class creation. For the
> special-casing to happen, the __new__ method has to be present when
> the class is created. If it is, then it automatically gets wrapped in
> a staticmethod. In the OP's case, he was adding the __new__ method
> after class creation, so the wrapping did not happen automatically.

Hmm, thank you. After trying a couple of things, it appears that the
reason OP's method worked in Python 3 is that __get__ on ordinary
functions in Python 3 simply returns the function itself when called on
type lookup, rather than Python 2's unbound method objects.

> http://docs.python.org/reference/dat...type-hierarchy
> """
> Class Types
> Class types, or “new-style classes,” are callable. These objects
> normally act as factories for new instances of themselves, but
> variations are possible for class types that override __new__(). The
> arguments of the call are passed to __new__() and, in the typical
> case, to __init__() to initialize the new instance.
> """
>
> AFAIK, that's pretty much it. When a type is called, __new__ is
> called to create the new instance, and then __init__ is called to
> initialize it (if __new__ returned an instance of the type).

Since that description doesn't include when __dict__ is created, it isn't
complete, however. I guess it's safe to assume that __dict__ is created
inside object.__new__, but that also leaves some things unclear; see
below.

>> how methods, properties and the like are bound;
>
> When they're accessed, using the descriptor protocol, not as part of
> the instantiation process. See:
> http://docs.python.org/reference/dat...ng-descriptors

Ah, sorry. I had read that, but I had mistakenly thought that the same
instance method object was always returned, which would have implied that
it had to be stored for the instance somewhere, at some point. I see now
that that is not actually the case, however. That clears up a whole lot
for me. Thanks!

>> when and whether __dict__
>> * is created and a couple of other things.
>
> Under the hood as part of the object creation process, unless the
> class uses __slots__.

And also unless the object created is of type `object' or any other
built-in type, which leaves me wondering exactly under what circumstances
__dict__ actually is created. Is it some kind of special case in
object.__new__ for types created with the `class' statement? There also
appear to be other special cases in object.__new__:

>>> object.__new__(dict)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: object.__new__(dict) is not safe, use dict.__new__()

I guess my question reduces into the following: What does object.__new__
actually do, and what other special cases does it implement?

--

Fredrik Tolf

On Tue, Dec 27, 2011 at 10:41 AM, K Richard Pixley <> wrote:
> The conceptual leap for me was in recognizing that a class is just an
> object. *The best way, (imo, so far), to create a singleton in python is to
> use the class itself as the singleton rather than ever instantiating it.
> *With a little work, you can even prevent it from ever being instantiated.

I don't think that's actually possible:

Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
(Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> class Foo:
.... def __new__(cls):
.... raise TypeError
.... def __init__(self):
.... raise TypeError
....
>>> type(object.__new__(Foo))
<class '__main__.Foo'>

I prefer the Borg pattern for singletons myself. When you need a Borg
instance, you just instantiate the class normally, do whatever you
need with it, and then discard it normally. It doesn't matter how
many instances there are (there can even be zero) because they all
share identical state. It's better encapsulation too, since client
code does not even need to know that the class is a singleton variant;
that becomes an implementation detail.

http://www.aleax.it/Python/5ep.html
http://code.activestate.com/recipes/66531/


> But __metaclass__ could use a few more examples, imo. *I'm still not
> entirely clear on __metaclass__. *As I understand it, (which may well be
> completely wrong), __metaclass_ is the means of perturbing the results of
> "isinstance". *You can create, (or delete), inheritance arbitrarily, and
> without even requiring real, existing classes to do it.

A metaclass is the class of another class (which is normally `type` if
no metaclass is specified). It's used to customize the way that the
class is created, in the same way that the class can be used to
customize the way instances are created (e.g. by overriding __new__).
You could use it to mess with the class's base classes, although I
think that usage is rare. It's more common to use it to automatically
add or modify entries in the class dict. For example, Django models
use a metaclass to collect all the fields and options declared for the
model class and to setup a default manager for the model class.

Note that many things that can be done with a metaclass can also be
done with a class decorator. The main difference as I see it is that
the metaclass is inherited by subclasses, whereas a decorator would
need to be applied individually to each subclass.

On 12/27/11 10:28 , Ian Kelly wrote:
> On Tue, Dec 27, 2011 at 10:41 AM, K Richard Pixley<> wrote:
>> The conceptual leap for me was in recognizing that a class is just an
>> object. The best way, (imo, so far), to create a singleton in python is to
>> use the class itself as the singleton rather than ever instantiating it.
>> With a little work, you can even prevent it from ever being instantiated.
> I don't think that's actually possible:
>
> Python 3.2 (r32:88445, Feb 20 2011, 21:29:02) [MSC v.1500 32 bit
> (Intel)] on win32
> Type "help", "copyright", "credits" or "license" for more information.
>>>> class Foo:
> ... def __new__(cls):
> ... raise TypeError
> ... def __init__(self):
> ... raise TypeError
> ...
>>>> type(object.__new__(Foo))
> <class '__main__.Foo'>

Try:

class Foo(object):
def __new__(cls):
return cls

--rich