Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Please explain the meaning of 'stealing' a ref

Reply
Thread Tools

Please explain the meaning of 'stealing' a ref

 
 
Christos TZOTZIOY Georgiou
Guest
Posts: n/a
 
      11-04-2003
I invested some time today because I got this crazy idea that if I
implemented a simple attribute-getting caching scheme implemented in
PyObject_GetAttr / PyObject_SetAttr that would turn out to be good.

So I did implement one touching Objects/object.c (and pythonrun.c for
init/final purposes), and I got a 5% increase in pystone... of course,
when I exited the interpreter, I got a nice segfault Also,
test.testall segfaulted at test_anydbm. I didn't expect anything
better, to be honest; it's obviously got something to do with inc/dec
refs.

Now I'm googling for the various debug building options (I'm not sure
that -DDEBUG is enough etc), however I realised it's time to ask what I
never understood so far: what exactly is the meaning of a function
"stealing a reference"? Is it that the function does an incref or a
decref? Or is it something more obscure?

TIA for any answers.

PS Do we have still time to help Guido be the one that throws the pie?-)
--
TZOTZIOY, I speak England very best,
Ils sont fous ces Redmontains! --Harddix
 
Reply With Quote
 
 
 
 
KefX
Guest
Posts: n/a
 
      11-04-2003
> what exactly is the meaning of a function
>"stealing a reference"?


It decrements the reference count. If you still need the reference, Py_INCREF()
it beforehand.

- Kef

 
Reply With Quote
 
 
 
 
Edward C. Jones
Guest
Posts: n/a
 
      11-04-2003
Christos TZOTZIOY Georgiou wrote:

> ... what exactly is the meaning of a function
> "stealing a reference"?


I have never liked teh "stealing" metaphor. My attempt to explain this
is at http://members.tripod.com/~edcjones/refcount.html

 
Reply With Quote
 
Michael Hudson
Guest
Posts: n/a
 
      11-04-2003
Christos "TZOTZIOY" Georgiou <(E-Mail Removed)> writes:

> I invested some time today because I got this crazy idea that if I
> implemented a simple attribute-getting caching scheme implemented in
> PyObject_GetAttr / PyObject_SetAttr that would turn out to be good.


Are you aware of the cache-attr-branch (think it's called that) in
CVS?

> So I did implement one touching Objects/object.c (and pythonrun.c for
> init/final purposes), and I got a 5% increase in pystone... of course,
> when I exited the interpreter, I got a nice segfault Also,
> test.testall segfaulted at test_anydbm. I didn't expect anything
> better, to be honest; it's obviously got something to do with inc/dec
> refs.
>
> Now I'm googling for the various debug building options (I'm not sure
> that -DDEBUG is enough etc), however I realised it's time to ask what I
> never understood so far: what exactly is the meaning of a function
> "stealing a reference"? Is it that the function does an incref or a
> decref? Or is it something more obscure?


This *really* should be explain in the API reference or the extended
and embedding manual somewhere... have you looked there?

Cheers,
mwh

--
There are two ways of constructing a software design: one way is to
make it so simple that there are obviously no deficiencies and the
other way is to make it so complicated that there are no obvious
deficiencies. -- C. A. R. Hoare
 
Reply With Quote
 
Christos TZOTZIOY Georgiou
Guest
Posts: n/a
 
      11-04-2003
On Tue, 4 Nov 2003 12:32:22 GMT, rumours say that Michael Hudson
<(E-Mail Removed)> might have written:

>Christos "TZOTZIOY" Georgiou <(E-Mail Removed)> writes:
>

[my attempting to implement attribute caching]
>
>Are you aware of the cache-attr-branch (think it's called that) in
>CVS?


<sigh> Obviously, not

[my question about what "stealing means"]
>
>This *really* should be explain in the API reference or the extended
>and embedding manual somewhere... have you looked there?


Oh, definitely, ext/refcountsInPython.html is the page I believe. I can
understand the second paragraph, but I wanted some reasoning. For
example, you call a function that its API specifies that it "steals" a
reference to its argument; therefore, you got to incref in advance and
decref afterwards yourself. What's the reason? Efficiency for
simplicity of the function? If yes, why not enclose the function call
in a incref / decref cycle and then export the enclosing function in the
API?

Such stuff I wanted to know.

Also, "borrowing" and "stealing" are the same thing? I just think that
"beautifying" terminology at the C level is more confusing than helpful.

PS Thanks KefX and Edward for your replies. I'll check the branch.
--
TZOTZIOY, I speak England very best,
Ils sont fous ces Redmontains! --Harddix
 
Reply With Quote
 
Alex Martelli
Guest
Posts: n/a
 
      11-04-2003
Christos TZOTZIOY Georgiou wrote:
...
> Oh, definitely, ext/refcountsInPython.html is the page I believe. I can
> understand the second paragraph, but I wanted some reasoning. For
> example, you call a function that its API specifies that it "steals" a
> reference to its argument; therefore, you got to incref in advance and


Yes, you have to incref IF you want to keep owning that reference.

> decref afterwards yourself. What's the reason? Efficiency for


You'll decref when you don't want to own that reference any more,
and you incref'd it, but if you do that soon, why bother incref'ing?-)

The typical use case to explain the stealing is in the "Reference
Count Details" section of the API manual:

PyObject *t;

t = PyTuple_New(3);
PyTuple_SetItem(t, 0, PyInt_FromLong(1L));
PyTuple_SetItem(t, 1, PyInt_FromLong(2L));
PyTuple_SetItem(t, 2, PyString_FromString("three"));


PyTuple_SetItem is very often called with a completely new reference
as its 3d argument, so it steals it to enable this easy idiom... w/o
the stealing you'd have to code, e.g.:

PyObject *t;
PyObject *temp;

t = PyTuple_New(3);

temp = PyInt_FromLong(1L);
PyTuple_SetItem(t, 0, temp);
Py_DECREF(temp);

etc -- much less handy.


> simplicity of the function? If yes, why not enclose the function call
> in a incref / decref cycle and then export the enclosing function in the
> API?


I don't know what you're talking about. Explain please? E.g. with one
of the few reference-stealing functions, PyTuple_SetItem ?


> Such stuff I wanted to know.
>
> Also, "borrowing" and "stealing" are the same thing? I just think that


No! You get a borrowed reference when you get a reference but are not
transferred the ownership of it. That's a pretty widespread use regarding
references that are returned to objects that surely pre-existed. You
must then incref in the relatively rare case you want to keep hold of
that reference for the longer term.

> "beautifying" terminology at the C level is more confusing than helpful.


What 'beautifying'...?


Alex

 
Reply With Quote
 
Alex Martelli
Guest
Posts: n/a
 
      11-04-2003
KefX wrote:

>> what exactly is the meaning of a function
>>"stealing a reference"?

>
> It decrements the reference count.


*NO*! It _takes over ownersip_: it DOESN'T decrement the
refcount at that point; rather, it takes over the future
responsibility for decrementing it at some point in time,
i.e., the abstract "ownership" of the reference. See my
other post in this thread for the classic example from Python's
online API docs ... if PyTuple_SetItem _decremented the
reference_ to the newly created PyObject* it's passed, how
WOULD it fill the tuple's item?-)

> If you still need the reference,
> Py_INCREF() it beforehand.


If you still need to OWN the reference, yes. If you just
need to USE it for a while during which you're sure the
tuple that's now "owning" it isn't going away, no need to
incref-then-decref.


Alex

 
Reply With Quote
 
Christos TZOTZIOY Georgiou
Guest
Posts: n/a
 
      11-05-2003
On Tue, 04 Nov 2003 14:20:18 GMT, rumours say that Alex Martelli
<(E-Mail Removed)> might have written:

I want to clarify that all the questions in my first post and in this
post are intended to help me understand the notions of "stealing a
reference", "borrowing a reference" because I found
api/refcountDetails.html a little unsatisfying.

[snip: my referring to ext/refcountsInPython.html and supposing that an
incref should be issued before calling a function "stealing" a
reference]

>Yes, you have to incref IF you want to keep owning that reference.


So, "owning a reference" means simply that some code did an incref and
is responsible to decref, right? This is my first request for
clarification, although I understand that so far I was confusing "owning
a reference" and "owning an object".

>> decref afterwards yourself. What's the reason? Efficiency for

>
>You'll decref when you don't want to own that reference any more,
>and you incref'd it, but if you do that soon, why bother incref'ing?-)


I'd do that because I did not understand what "stealing a reference"
means; therefore I translated "stealing" = decref, so I presumed that I
should issue an incref in advance in order to make sure the object
passed to the function does not cease to exist before the function
returns. I assumed based on unclear (to me) terms.

>The typical use case to explain the stealing is in the "Reference
>Count Details" section of the API manual:
>
>PyObject *t;
>
>t = PyTuple_New(3);
>PyTuple_SetItem(t, 0, PyInt_FromLong(1L));
>PyTuple_SetItem(t, 1, PyInt_FromLong(2L));
>PyTuple_SetItem(t, 2, PyString_FromString("three"));
>
>PyTuple_SetItem is very often called with a completely new reference
>as its 3d argument, so it steals it to enable this easy idiom... w/o
>the stealing you'd have to code, e.g.:
>
>PyObject *t;
>PyObject *temp;
>
>t = PyTuple_New(3);
>
>temp = PyInt_FromLong(1L);
>PyTuple_SetItem(t, 0, temp);
>Py_DECREF(temp);
>
>etc -- much less handy.


I understand that. So, a function "stealing" a reference means the
function takes away from the caller the responsibility for decref'ing
the object, and "stealing a reference" does not translate into some C
code, right? This is my second request for clarification.

>> simplicity of the function? If yes, why not enclose the function call
>> in a incref / decref cycle and then export the enclosing function in the
>> API?

>
>I don't know what you're talking about. Explain please? E.g. with one
>of the few reference-stealing functions, PyTuple_SetItem ?


You don't know what I'm talking about because neither I was sure what I
was talking about; I now see I should not make assumptions since I was
not sure as to what "stealing a ref" means. It was confusing to the
readers of my post.

>> Such stuff I wanted to know.
>>
>> Also, "borrowing" and "stealing" are the same thing? I just think that

>
>No! You get a borrowed reference when you get a reference but are not
>transferred the ownership of it. That's a pretty widespread use regarding
>references that are returned to objects that surely pre-existed. You
>must then incref in the relatively rare case you want to keep hold of
>that reference for the longer term.


So a function "borrowing" a reference means that the function does not
incref or decref an object which was passed to it, right? That is the
third request.
I assume that most functions of the python API "borrow" references?

>> "beautifying" terminology at the C level is more confusing than helpful.

>
>What 'beautifying'...?


"Beautifying" in the sense that I thought that both "stealing" and
"borrowing" a reference translate into actual C code, a mistaken
assumption.

Thanks for your time.
--
TZOTZIOY, I speak England very best,
Ils sont fous ces Redmontains! --Harddix
 
Reply With Quote
 
Alex Martelli
Guest
Posts: n/a
 
      11-05-2003
Christos TZOTZIOY Georgiou wrote:
...
>>Yes, you have to incref IF you want to keep owning that reference.

>
> So, "owning a reference" means simply that some code did an incref and
> is responsible to decref, right? This is my first request for


"Owning a reference" means "I'll dispose of that reference when I'm
done with it". As per:

http://www.python.org/doc/current/ap...ntDetails.html

"""
The reference count behavior of functions in the Python/C API is best
explained in terms of ownership of references. Note that we talk of owning
references, never of owning objects; objects are always shared! When a
function owns a reference, it has to dispose of it properly -- either by
passing ownership on (usually to its caller) or by calling Py_DECREF() or
Py_XDECREF(). When a function passes ownership of a reference on to its
caller, the caller is said to receive a new reference. When no ownership is
transferred, the caller is said to borrow the reference. Nothing needs to
be done for a borrowed reference.
"""

I really don't know how we could have expressed this any more clearly;
if you can indicate what is hard for you to understand here, and how
we could make it easier, I'd be really greateful!

> clarification, although I understand that so far I was confusing "owning
> a reference" and "owning an object".


Ditto: I don't know how we could try to dispell this common doubt
any more clearly than by saying "we talk of owning references, never
of owning objects; objects are always shared!".


>>> decref afterwards yourself. What's the reason? Efficiency for

>>
>>You'll decref when you don't want to own that reference any more,
>>and you incref'd it, but if you do that soon, why bother incref'ing?-)

>
> I'd do that because I did not understand what "stealing a reference"
> means; therefore I translated "stealing" = decref, so I presumed that I
> should issue an incref in advance in order to make sure the object
> passed to the function does not cease to exist before the function
> returns. I assumed based on unclear (to me) terms.


Hmmm -- wasn't the example on the same URL I just quoted, right after
the 2nd paragrap, clear enough in terms of no incref being needed?
Can you suggest a way in which we could make it clearer to you?


> I understand that. So, a function "stealing" a reference means the
> function takes away from the caller the responsibility for decref'ing
> the object, and "stealing a reference" does not translate into some C
> code, right? This is my second request for clarification.


Right. There is no specific C code to which you can point and say
"see, right here, the reference is being stolen". If you looked at
the source for e.g. PyTuple_SetItem you might notice it *doesn't*
incref (nor of course decref) its 3rd argument -- it copies that
PyObject* somewhere _and that's it_ -- contrasted with the "normal"
case of e.g. PySequence_SetItem which copies the PyObject* s/where
AND incref's it. So the "stealing" is the copying-somewhere plus
the LACK of the incref that would normally go with copying, in terms
of C source.


>>> Also, "borrowing" and "stealing" are the same thing? I just think that

>>
>>No! You get a borrowed reference when you get a reference but are not
>>transferred the ownership of it. That's a pretty widespread use regarding
>>references that are returned to objects that surely pre-existed. You
>>must then incref in the relatively rare case you want to keep hold of
>>that reference for the longer term.

>
> So a function "borrowing" a reference means that the function does not
> incref or decref an object which was passed to it, right? That is the
> third request.


The function will incref the reference passed to if IF it's copying
it somewhere (and will then arrange for that copy to be decref'd later
when that reference is not needed any more). If the function is
just using the reference and is done with it when it returns, then
the function doesn't bother incref'ing it. The caller keeps ownership
if it originally had it (i.e. unless it was borrowing from somewhere
else, then ownership remains with whoever ultimately had it), there
is no ownership transfer for that reference.

> I assume that most functions of the python API "borrow" references?


Yes, and so do all functions you'll be writing to be callable from
Python -- it's the normal case.

> Thanks for your time.


You're welcome! Now I'd like, if possible, to enhance the docs so
that other people's future doubts might be similarly helped -- that's
the reason I'm asking you for advice on this!

If you could spend some time to visit safari.oreilly.com -- subscribe
and be sure to cancel before 2 weeks so you don't have to pay! -- and
check the penultimate chapter of Python in a Nutshell, where I've
tried an alternate "concise mixed reference/tutorial" tack to the
whole issue of extending Python, I'd be particularly grateful of any
feedback about that, too... thanks!


Alex

 
Reply With Quote
 
Christos TZOTZIOY Georgiou
Guest
Posts: n/a
 
      11-05-2003
On Wed, 05 Nov 2003 15:07:08 GMT, rumours say that Alex Martelli
<(E-Mail Removed)> might have written:

>Christos TZOTZIOY Georgiou wrote:
> ...


[snip]

>"Owning a reference" means "I'll dispose of that reference when I'm
>done with it". As per:
>
>http://www.python.org/doc/current/ap...ntDetails.html
>
>"""
>The reference count behavior of functions in the Python/C API is best
>explained in terms of ownership of references. Note that we talk of owning
>references, never of owning objects; objects are always shared! When a
>function owns a reference, it has to dispose of it properly -- either by
>passing ownership on (usually to its caller) or by calling Py_DECREF() or
>Py_XDECREF(). When a function passes ownership of a reference on to its
>caller, the caller is said to receive a new reference. When no ownership is
>transferred, the caller is said to borrow the reference. Nothing needs to
>be done for a borrowed reference.
>"""
>
>I really don't know how we could have expressed this any more clearly;
>if you can indicate what is hard for you to understand here, and how
>we could make it easier, I'd be really greateful!


The missing part IMO is what "owning a reference" means. I understand
that my brain moved in mysterious ways on the false assumption that
"steal"/"borrow" translates to C code, yet I believe a definition of the
term "own" should be inserted somewhere in a way like:

"... objects are always shared! Every call to Py_INCREF creates a
'reference' to the object (actually, it's a way to make 'official' to
the python memory management system that you've got a pointer pointing
to the object, so the object better stays where it is!) which reference
is considered as 'owned' by the caller. This reference should be
disposed of properly ..."

I beieve that neither my english nor my brain is fit for documentation
writing, but I also believe you will get the idea. Perhaps the text in
the parenthesis needs a complete rewriting --or removal.
The definition could be appended as a footnote, to avoid changing the
current text flow.

>> clarification, although I understand that so far I was confusing "owning
>> a reference" and "owning an object".

>
>Ditto: I don't know how we could try to dispell this common doubt
>any more clearly than by saying "we talk of owning references, never
>of owning objects; objects are always shared!".


This text is obviously clear, and the misconception was my fault: since
references are just pointer variables at the C level, and I never had a
concept of a function (or a program) 'owning' its variables,
automatically my brain converted the word 'reference' to 'reference to
python object' to 'python object'. That's it.

[snip]

>You're welcome! Now I'd like, if possible, to enhance the docs so
>that other people's future doubts might be similarly helped -- that's
>the reason I'm asking you for advice on this!


I proposed a definition for "owning a reference".

A slight clarification in the following text:
"""Few functions steal references; the two notable exceptions are
PyList_SetItem()and PyTuple_SetItem()"""
The word "exceptions" implies "exception to the previous statement".
IIUC "exceptions" should be changed to "examples".

Also, it would be helpful if (possibly as footnotes too) there was the
following definition:
"stealing a reference": a function that steals a reference takes away
from the caller the responsibility of calling Py_DECREF on the
referenced object.

If all of the above seem redundant to you, then perhaps it would make
life easier for future readers adding a link to ext/ownershipRules.html
at the end of the first sentence of api/refcountDetails.html . (as in
HTML lingo 'see also <A href="/ext/ownershipRules.html">Ownership
Rules</A>')

Borrowing and reference ownership are explained in 'Ownership Rules',
which is in the Extending and Embedding tutorial; one tends to think
that a reference manual should not depend on definitions in a
tutorial...

>If you could spend some time to visit safari.oreilly.com -- subscribe
>and be sure to cancel before 2 weeks so you don't have to pay! -- and
>check the penultimate chapter of Python in a Nutshell, where I've
>tried an alternate "concise mixed reference/tutorial" tack to the
>whole issue of extending Python, I'd be particularly grateful of any
>feedback about that, too... thanks!


Thanks for the hint, I'll try that sometime.
--
TZOTZIOY, I speak England very best,
Ils sont fous ces Redmontains! --Harddix
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Giving an rvalue ref to a function taking an rvalue ref Juha Nieminen C++ 13 08-29-2012 09:25 PM
passing ref to ptr again as ref to ptr.... osama178@gmail.com C++ 22 04-30-2008 10:42 AM
passing ref to ptr again as ref to ptr.... osama178@gmail.com C++ 6 04-29-2008 08:09 AM
passing ref to ptr again as ref to ptr.... osama178@gmail.com C++ 0 04-24-2008 08:23 PM
strong ref from weak ref? Navindra Umanee Ruby 2 02-12-2005 05:06 PM



Advertisments