Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Cyclic GC rules for subtyped objects with tp_dictoffset

Reply
Thread Tools

Cyclic GC rules for subtyped objects with tp_dictoffset

 
 
BChess
Guest
Posts: n/a
 
      03-31-2009
Hi,

I'm writing a new PyTypeObject that is base type, supports cyclic GC,
and has a tp_dictoffset. If my type is sub-typed by a python class,
what exactly are the rules for how I'm supposed to treat my PyDict
object with regards to cyclic GC? Do I still visit it in my traverse
() function if I'm subtyped? Do I decrement the refcount upon
dealloc? By the documentation, I'm assuming I should always be using
_PyObject_GetDictPtr() to be accessing the dictionary, which I do.
But visiting the dictionary in traverse() in the case it's subtyped
results in a crash in weakrefobject.c. I'm using Python 2.5.

Thanks for any help anyone has!
Ben
 
Reply With Quote
 
 
 
 
Hrvoje Niksic
Guest
Posts: n/a
 
      03-31-2009
[ Questions such as this might be better suited for the capi-sig list,
http://mail.python.org/mailman/listinfo/capi-sig ]

BChess <(E-Mail Removed)> writes:

> I'm writing a new PyTypeObject that is base type, supports cyclic
> GC, and has a tp_dictoffset. If my type is sub-typed by a python
> class, what exactly are the rules for how I'm supposed to treat my
> PyDict object with regards to cyclic GC? Do I still visit it in my
> traverse () function if I'm subtyped? Do I decrement the refcount
> upon dealloc? By the documentation, I'm assuming I should always be
> using _PyObject_GetDictPtr() to be accessing the dictionary, which I
> do. But visiting the dictionary in traverse() in the case it's
> subtyped results in a crash in weakrefobject.c. I'm using Python
> 2.5.


First off, if your class is intended only as a base class, are you
aware that simply inheriting from a dictless class adds a dict
automatically? For example, the base "object" type has no dict, but
inheriting from it automatically adds one (unless you override that
using __slots__). Having said that, I'll assume that the base class
is usable on its own and its direct instances need to have a dict as
well.

I'm not sure if this kind of detail is explicitly documented, but as
far as the implementation goes, the answer to your question is in
Objects/typeobject.c:subtype_traverse. That function gets called to
traverse instances of heap types (python subclasses of built-in
classes such as yours). It contains code like this:

if (type->tp_dictoffset != base->tp_dictoffset) {
PyObject **dictptr = _PyObject_GetDictPtr(self);
if (dictptr && *dictptr)
Py_VISIT(*dictptr);
}

According to this, the base class is responsible for visiting its dict
in its tp_traverse, and the subtype only visits the dict it added
(which is why its location differs). Note that visiting an object
twice still shouldn't cause a crash; objects may be and are visited an
arbitrary number of times, and it's up to the GC to ignore those it
has already seen. So it's possible that you have a bug elsewhere in
the code.

As far as the decrementing goes, the rule of thumb is: if you created
it, you get to decref it. subtype_dealloc contains very similar
logic:

/* If we added a dict, DECREF it */
if (type->tp_dictoffset && !base->tp_dictoffset) {
PyObject **dictptr = _PyObject_GetDictPtr(self);
if (dictptr != NULL) {
PyObject *dict = *dictptr;
if (dict != NULL) {
Py_DECREF(dict);
*dictptr = NULL;
}
}
}

So, if the subtype added a dict, it was responsible for creating it
and it will decref it. If the dict was created by you, it's up to you
to dispose of it.
 
Reply With Quote
 
 
 
 
BChess
Guest
Posts: n/a
 
      03-31-2009
On Mar 31, 12:27*am, Hrvoje Niksic <(E-Mail Removed)> wrote:
> [ Questions such as this might be better suited for the capi-sig list,
> *http://mail.python.org/mailman/listinfo/capi-sig]
>
> BChess<(E-Mail Removed)> writes:
> > I'm writing a new PyTypeObject that is base type, supports cyclic
> > GC, and has a tp_dictoffset. *If my type is sub-typed by a python
> > class, what exactly are the rules for how I'm supposed to treat my
> > PyDict object with regards to cyclic GC? *Do I still visit it in my
> > traverse () function if I'm subtyped? *Do I decrement the refcount
> > upon dealloc? *By the documentation, I'm assuming I should always be
> > using _PyObject_GetDictPtr() to be accessing the dictionary, which I
> > do. *But visiting the dictionary in traverse() in the case it's
> > subtyped results in a crash in weakrefobject.c. *I'm using Python
> > 2.5.

>
> First off, if your class is intended only as a base class, are you
> aware that simply inheriting from a dictless class adds a dict
> automatically? *For example, the base "object" type has no dict, but
> inheriting from it automatically adds one (unless you override that
> using __slots__). *Having said that, I'll assume that the base class
> is usable on its own and its direct instances need to have a dict as
> well.
>
> I'm not sure if this kind of detail is explicitly documented, but as
> far as the implementation goes, the answer to your question is in
> Objects/typeobject.c:subtype_traverse. *That function gets called to
> traverse instances of heap types (python subclasses of built-in
> classes such as yours). *It contains code like this:
>
> * * *if (type->tp_dictoffset != base->tp_dictoffset) {
> * * * * *PyObject **dictptr = _PyObject_GetDictPtr(self);
> * * * * * * *if (dictptr && *dictptr)
> * * * * * * * * *Py_VISIT(*dictptr);
> * * *}
>
> According to this, the base class is responsible for visiting its dict
> in its tp_traverse, and the subtype only visits the dict it added
> (which is why its location differs). *Note that visiting an object
> twice still shouldn't cause a crash; objects may be and are visited an
> arbitrary number of times, and it's up to the GC to ignore those it
> has already seen. *So it's possible that you have a bug elsewhere in
> the code.
>
> As far as the decrementing goes, the rule of thumb is: if you created
> it, you get to decref it. *subtype_dealloc contains very similar
> logic:
>
> * * * * /* If we added a dict, DECREF it */
> * * * * if (type->tp_dictoffset && !base->tp_dictoffset) {
> * * * * * * * * PyObject **dictptr = _PyObject_GetDictPtr(self);
> * * * * * * * * if (dictptr != NULL) {
> * * * * * * * * * * * * PyObject *dict = *dictptr;
> * * * * * * * * * * * * if (dict != NULL) {
> * * * * * * * * * * * * * * * * Py_DECREF(dict);
> * * * * * * * * * * * * * * * * *dictptr = NULL;
> * * * * * * * * * * * * }
> * * * * * * * * }
> * * * * }
>
> So, if the subtype added a dict, it was responsible for creating it
> and it will decref it. *If the dict was created by you, it's up to you
> to dispose of it.


My confusion stemmed from the fact that I wasn't actually,
technically, allocating a PyDict in this space.
PyObject_GenericSetAttr() does that automatically when it finds that
it's NULL. But that seems to be the same as if I had made it myself
-- so I'm to dealloc either way.

Thank you for the very in-depth answer, though. You're right: the
problem was elsewhere. The crash stemmed from using a negative number
for tp_dictoffset. This doesn't seem to do the right thing when
subtyping -- the tp_dictoffset was pointing to the same memory as the
offset specified in the subtype's tp_weaklistoffset. I missed the
sentence in the documentation that negative tp_dictoffsets should only
be used for variable-length objects. Using a positive offset instead
worked like a charm.

Thanks again,
Ben
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
C-API, tp_dictoffset vs tp_members Ulrich Eckhardt Python 5 07-26-2009 09:24 PM
class objects, method objects, function objects 7stud Python 11 03-20-2007 06:05 PM
rules for Cisco PIX 525 firewall rules KAS Cisco 2 10-02-2005 07:12 PM
Can an object of a C type have random members added? tp_dictoffset? Gregory Bond Python 1 04-21-2005 03:19 PM
Cyclic object graph question Laird Nelson Java 2 10-10-2003 07:53 PM



Advertisments