Messing with the GC
 

Hi,

triggered by some problems I had with PySide I got a bit
confused about what the GC may do in certain situations.
Here's a small test program I cobbled together:

import sys

class X( object ) :
def __init__( self, parent, cnt ) :
print( "In constructor for {0} {1}".format( self, cnt ),
file = sys.stderr )
self.parent = parent
self.cnt = cnt

def __del__( self ) :
print( "In destructor for {0} {1}".format( self, self.cnt ),
file = sys.stderr )

def foo( self ) :
print( "Before", file = sys.stderr )
self.parent.z = X( self.parent, 2 ) # Is this bad?
print( "After", file = sys.stderr )

class Y( object ) :
def __init__( self ) :
print( "In constructor for {0}".format( self ),
file = sys.stderr )
self.z = X( self, 1 )

def __del__( self ) :
print( "In destructor for {0}".format( self ),
file = sys.stderr )

Y( ).z.foo( )

Have a look at the line with the comment. At this point the
only reference in existence to the X class instance, of which
a method is just being executed, goes out of scope. Thus I
would assume that the GC could now kick any time, possibly
even before the following call of print() or before the method
call returns. That, in turn might result in a crash of the
script.

Is my assumption about this flawed and there are no potential
dangers? Perhaps with

Y( ).z.foo( )

a temporary second reference is created that keeps the GC
for removing the X instance...

Another thing I'm puzzled about is the output of the
script:

In constructor for <__main__.Y object at 0x2919210>
In constructor for <__main__.X object at 0x2919310> 1
Before
In constructor for <__main__.X object at 0x2919350> 2
After
In destructor for <__main__.X object at 0x2919310> 1

Ok, the destrucor for the first instance of the X class is
called only after printing out "After", so the GC didn't
delete the object before. But then there are obviously no
calls of the destructors of neither the second instance
of the X class nor of the Y class instance. Shouldn't
they be invoked before the program ends?

Thanks and best regards, Jens
--
\ Jens Thoms Toerring ___ jt@toerring.de
\__________________________ http://toerring.de

Re: Messing with the GC
 

On Sat, 19 Jan 2013 14:47:16 +0000, Jens Thoms Toerring wrote:

> Ok, the destrucor for the first instance of the X class is called only
> after printing out "After", so the GC didn't delete the object before.
> But then there are obviously no calls of the destructors of neither the
> second instance of the X class nor of the Y class instance. Shouldn't
> they be invoked before the program ends?

You should avoid __del__ destructors whenever not absolutely necessary.

__del__ may not be called for objects that still exist when Python exits.

If you have a cycle of objects, and *any* of them have a __del__ method,
it may be impossible for Python to work out a safe order for them to be
deleted. Consequently they will never be reclaimed by the garbage
collector.

http://docs.python.org/2/reference/d...l-method-names

http://docs.python.org/2/library/gc.html#gc.garbage



--
Steven

Re: Messing with the GC
 

On Sat, 19 Jan 2013 16:24:37 +0000, Steven D'Aprano wrote:

> On Sat, 19 Jan 2013 14:47:16 +0000, Jens Thoms Toerring wrote:
>
>> Ok, the destrucor for the first instance of the X class is called only
>> after printing out "After", so the GC didn't delete the object before.
>> But then there are obviously no calls of the destructors of neither the
>> second instance of the X class nor of the Y class instance. Shouldn't
>> they be invoked before the program ends?
>
> You should avoid __del__ destructors whenever not absolutely necessary.

And here is another opinion: you should avoid cycles, rather the __del__.

http://eli.thegreenplace.net/2009/06...ors-in-python/





--
Steven

Re: Messing with the GC
 

On 1/19/2013 9:47 AM, Jens Thoms Toerring wrote:

The code comments mostly answer your questions about what happens or
does not happen and when. The comments after add more detail.

> import sys
>
> class X( object ) :
> def __init__( self, parent, cnt ) :
> print( "In constructor for {0} {1}".format( self, cnt ),
> file = sys.stderr )
> self.parent = parent
> self.cnt = cnt
>
> def __del__( self ) :
> print( "In destructor for {0} {1}".format( self, self.cnt ),
> file = sys.stderr )
>
> def foo( self ) :

At this point, self is self.parent.z, which is to say, self and
self.parent.z are 2 references to 1 object. The self and self.parent
object refer to each other and therefore constitute a reference cycle.

> print( "Before", file = sys.stderr )
> self.parent.z = X( self.parent, 2 ) # Is this bad?

At this point, self.parent.z is another instance of X, breaking the
existing reference cycle *and* creating a new one. Now self has just the
one reference 'self' (plus another non-circular, hidden one, see below#).

> print( "After", file = sys.stderr )

At this point, self goes out of scope, the CPython reference count goes
to 0, and the object that was self is deleted. Other implementations
typically wait longer to delete.

> class Y( object ) :
> def __init__( self ) :
> print( "In constructor for {0}".format( self ),
> file = sys.stderr )
> self.z = X( self, 1 )

At this point, self is self.z.parent, where z is an X. This is a
circular reference: self and self.z reference each other.

> def __del__( self ) :
> print( "In destructor for {0}".format( self ),
> file = sys.stderr )
>
> Y( ).z.foo( )

Y() creates a reference cycle. z.foo() substitute a different instance
of X in the cycle but there still is a cycle, so the Y and X objects
have reference counts of 1. There are no other references to the two
objects, making them unreachable and 'dead' to the program.

The cyclic reference garbage collector (see the gc module doc) that is
meant to delete such orphans does not delete them here because of the
__del__ methods. Since gc was added, __del__ is semi-obsolete. If an
object might possibly be put in a reference cycle (which is quite easy),
any code that might have been put in __del__ should go in an explicitly
called .close() method.

> Have a look at the line with the comment. At this point the
> only reference in existence to the X class instance, of which
> a method is just being executed, goes out of scope.

Nope, the remaining reference, 'self', stays in scope until after the
function exits. That is when X1 is deleted and the deletion message
printed.

> Is my assumption about this flawed

Yes

> and there are no potential dangers?

The orphaned two-object cycle constitutes a memory leak.
If you called Y( ).z.foo( ) a million times,
you would have a million useless pairs of objects.
This is why gc was added.

> Y( ).z.foo( )
>
> [perhaps] a temporary second reference is created that keeps the GC
> for removing the X instance...

Function calls (normally) bind argument object to parameter names in the
function's local namespace. That binding is a temporary reference and
objects will not disappear in the middle of the call.

# In CPython, at least, there is another internal reference to arguments
that also disappears when the function returns, allowing the deletion of
arguments without other references.

>>> x = 12343252 # random large number to make sure its a new object
>>> sys.getrefcount(x)
2
>>> def f(a): print(sys.getrefcount(a))

>>> f(x)
4

So print(sys.getrefcount(self)) at the top of foo prints 4 (3+1),
rather than 3 (2+1), as one might expect. The +1 is explained in the doc

sys.getrefcount(obj):
Return the reference count of the object. The count returned is
generally one higher than you might expect, because it includes the
(temporary) reference as an argument to getrefcount().

(Why doesn't getrefcount add 2 instead of 1, as f seems to? I don't
know, but perhaps because it is written in C rather than Python and
Python code objects are different from C code.)

--
Terry Jan Reedy

Re: Messing with the GC
 

And further thoughts...

On Sat, 19 Jan 2013 14:47:16 +0000, Jens Thoms Toerring wrote:

> Hi,
>
> triggered by some problems I had with PySide I got a bit
> confused about what the GC may do in certain situations. Here's a small
> test program I cobbled together:
>
> import sys
>
> class X( object ) :
> def __init__( self, parent, cnt ) :
> print( "In constructor for {0} {1}".format( self, cnt ),
> file = sys.stderr )
> self.parent = parent
> self.cnt = cnt
>
> def __del__( self ) :
> print( "In destructor for {0} {1}".format( self, self.cnt ),
> file = sys.stderr )
>
> def foo( self ) :
> print( "Before", file = sys.stderr )
> self.parent.z = X( self.parent, 2 ) # Is this bad?
> print( "After", file = sys.stderr )
>
> class Y( object ) :
> def __init__( self ) :
> print( "In constructor for {0}".format( self ),
> file = sys.stderr )
> self.z = X( self, 1 )
>
> def __del__( self ) :
> print( "In destructor for {0}".format( self ),
> file = sys.stderr )
>
> Y( ).z.foo( )
>
> Have a look at the line with the comment.


You mean this line?

self.parent.z = X( self.parent, 2 ) # Is this bad?


> At this point the only
> reference in existence to the X class instance, of which a method is
> just being executed, goes out of scope.

I don't understand this, but to the extent that I do understand it, I
think you are wrong.

What do you mean, "the X class instance"? If you mean the class X itself,
no, that survives until both the class itself is deleted and every one of
it's instances. If you mean "self", no, that doesn't get deleted by that
line at all.


> Thus I would assume that the GC
> could now kick any time, possibly even before the following call of
> print() or before the method call returns. That, in turn might result in
> a crash of the script.

It would be a pretty crappy garbage collector that collected objects
while they were still being used.



> Is my assumption about this flawed and there are no potential dangers?
> Perhaps with
>
> Y( ).z.foo( )
>
> a temporary second reference is created that keeps the GC for removing
> the X instance...

I'm not even sure what X instance you are referring to, or why you think
it is going out of scope.



--
Steven

Re: Messing with the GC
 

Hi,

thank you for the explanations. I had overlooked the
cyclic nature of what I had produced here and, of course,
the GC can't be blamed for not collecting objects that are
part of a cycle. The other question about the last refe-
rence to an object vanishing within a method call (which,
as I now clearly understand, can't happen and wouldn't make
much sense) was triggered by a segmentation fault I get
when I do something similar in PySide, so I was getting
worried if it might be due to a GC issue. Now I know its
got to be something different;-)

Thanks and best regards, Jens
--
\ Jens Thoms Toerring ___ jt@toerring.de
\__________________________ http://toerring.de

Re: Messing with the GC
 

On 1/20/2013 3:09 PM, Jens Thoms Toerring wrote:

> thank you for the explanations. I had overlooked the
> cyclic nature of what I had produced here and, of course,
> the GC can't be blamed for not collecting objects that are
> part of a cycle. The other question about the last refe-
> rence to an object vanishing within a method call (which,
> as I now clearly understand, can't happen and wouldn't make
> much sense) was triggered by a segmentation fault I get
> when I do something similar in PySide, so I was getting
> worried if it might be due to a GC issue. Now I know its
> got to be something different;-)

Perhaps the hardest part of writing C extensions to CPython directly in
C (versus something like Cython) is properly balancing increfs and
decrefs. An incref without a later decref can lead to a memory leak. A
decref without a preceding incref (so CPython thinks the object can be
deleted, when it should not be) can lead to segfaults. So I would report
PySide code leading to segfaults to the PySide people.

--
Terry Jan Reedy

Re: Messing with the GC
 

Terry Reedy <tjreedy@udel.edu> wrote:
> On 1/20/2013 3:09 PM, Jens Thoms Toerring wrote:

> > thank you for the explanations. I had overlooked the
> > cyclic nature of what I had produced here and, of course,
> > the GC can't be blamed for not collecting objects that are
> > part of a cycle. The other question about the last refe-
> > rence to an object vanishing within a method call (which,
> > as I now clearly understand, can't happen and wouldn't make
> > much sense) was triggered by a segmentation fault I get
> > when I do something similar in PySide, so I was getting
> > worried if it might be due to a GC issue. Now I know its
> > got to be something different;-)

> Perhaps the hardest part of writing C extensions to CPython directly in
> C (versus something like Cython) is properly balancing increfs and
> decrefs. An incref without a later decref can lead to a memory leak. A
> decref without a preceding incref (so CPython thinks the object can be
> deleted, when it should not be) can lead to segfaults.

Definitely - I got started with Python having to write glue
code to get Python to work with a C++ library. And keeping
track of which side thinks it owns an object can sometimes
be a bit of a challenge...

> So I would report PySide code leading to segfaults to the
> PySide people.

Now that I'm more sure that it's unlikely to be a Python GC
related issue (or my not understanding what I'm doing, to be
precise) this is on my to-do list. But first I have to distill
things down to a very short example program still exhibiting
the problem - and experience tells me that this will most li-
kely result in the realization that it's not a PySide issue
at all but some misunderstanding on my side;-)

Best regards, Jens
--
\ Jens Thoms Toerring ___ jt@toerring.de
\__________________________ http://toerring.de