Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Function decorator that caches function results

Reply
Thread Tools

Function decorator that caches function results

 
 
Steven D'Aprano
Guest
Posts: n/a
 
      10-09-2005
On Sun, 09 Oct 2005 14:27:32 +0200, Fredrik Lundh wrote:

> Steven D'Aprano wrote:
>
>> Yes I did. Did you read my post?

>
> given that the wikipedia page says
>
> "a closure is an abstraction representing a function, plus the
> lexical environment (see static scoping) in which the function
> was created."
>
> and you're still focussing on type(function), it sure looks as if you missed
> certain parts of that explanation.
>
> let's take it again, with emphasis on some important words:
>
> "a closure is an ABSTRACTION representing a function, PLUS the
> lexical ENVIRONMENT (see static scoping) in which the function
> was created."
>
>> If you create a closure, using a memoization technique as per the original
>> post, and then call type() on that closure, Python reports <type 'function'>.

>
> that's because "closure" is an abstract concept.


So are functions, modules, classes, instances, ints and strings.

> there is no "closure" object
> in Python, just as there is no "program" object.


Clearly there is no DISTINCT closure object. If there were, I wouldn't
need to ask how one can tell them apart, because type() would just report
that one was a function and one was a closure. I don't have a problem with
that. But read on...

> function objects always con-
> tain all the information they need about their context, and the sum of that is
> what forms the closure.


If what you say is true, then all functions are closures, and closure is
just a synonym for function, and there is no difference between a function
and a closure. Then why bother to create the term? Clearly, whoever
invented the term did so to distinguish the two.

At a practical, Python level, there is a difference between a function
before and after it gets made into a closure using, e.g. the original
poster's memoization technique. In Python at least, it is not true that a
function and a function-turned-into-closure is the same thing.

See, for example, this:-

>>> def closefy(fn):

.... def do_nothing(*args, **kwargs):
.... return fn(*args, **kwargs)
.... return do_nothing
....
>>> def spam1():

.... return 0
....
>>> spam2 = closefy(spam1)
>>> spam1()

0
>>> spam2()

0
>>> spam2 is spam1

False
>>> spam1.func_closure is None

True
>>> spam2.func_closure

(<cell at 0xf70a2734: function object at 0xf6e0a144>,)
>>> hex(id(spam1))

'0xf6e0a144'

In one case, the "raw" function has None stored in its func_closure
attribute. In the other, it has a tuple containing at least one object of
type cell. That cell object appears to contain a reference back to the
original "raw" function.

Now, I'll tell you what I think is going on: spam1() obviously has lexical
scope, but that scope doesn't need to be saved with the function because
it is implicitly understood by Python.

The output of closefy(), on the other hand, has scope different from
the "normal" Python scope. So its output has to package up enough
information to re-create it's lexical scope in a cell object, and
that gets stored in the output's func_closure attribute.

So no, there is no "abstract" difference between a closure and a raw
function, but there is a practical difference. Now perhaps in some other
languages there is no practical difference either, but (as far as I can
see) Python is not that language.

Am I close?

Here are another two code snippets that show something of what Python is
doing:-

>>> def thingo():

.... def shrubbery():
.... return 0
.... return shrubbery
....
>>> spam3 = thingo()
>>> spam3()

0
>>> spam3.func_closure is None

True

>>> def thingy():

.... n = 0
.... def shrubbery():
.... return n
.... return shrubbery
....
>>> spam4 = thingy()
>>> spam4.func_closure

(<cell at 0xf70a2b24: int object at 0x901a158>,)


--
Steven.

 
Reply With Quote
 
 
 
 
Fredrik Lundh
Guest
Posts: n/a
 
      10-09-2005
Ron Adam wrote:

> In effect, 'cache' and 'fn' are replaced by the objects they reference
> before the cached_result function is returned.


not really; accesses to "free variables" always go via special cell objects
(rather than direct object references), and the compiler uses special byte
codes for all accesses to such objects.

this snippet illustrates the differences:

def func1():
return a

def func2(a):
return a

def wrapper(a):
def func3():
return a
return func3

def decode(func):
import dis
print func.__name__ + ":"
dis.dis(func)
print "co_freevars =", func.func_code.co_freevars
print "func_closure =", func.func_closure
print

decode(func1)
decode(func2)
decode(wrapper(10))

if you run this on a recent version of Python, you get something like:

func1:
2 0 LOAD_GLOBAL 0 (a)
3 RETURN_VALUE
co_freevars = ()
func_closure = None

func2:
5 0 LOAD_FAST 0 (a)
3 RETURN_VALUE
co_freevars = ()
func_closure = None

func3:
9 0 LOAD_DEREF 0 (a)
3 RETURN_VALUE
co_freevars = ('a',)
func_closure = (<cell at 0x0092C970: int object at 0x008A537C>,)

note the use of LOAD_DEREF for the variable access.

Other opcodes include STORE_DEREF (that updates a local variable
through a cell, used inside the "defining" scope), and MAKE_CLOSURE
(which sets up the function object for functions that actually uses
nested scopes; this is where the func_closure attribute is initialized).

(now, if I weren't limited to plain text, I could have drawn a nice little
diagram for you, but that's an entirely different thread...)

</F>



 
Reply With Quote
 
 
 
 
Fredrik Lundh
Guest
Posts: n/a
 
      10-09-2005
Steven D'Aprano wrote:

> > let's take it again, with emphasis on some important words:
> >
> > "a closure is an ABSTRACTION representing a function, PLUS the
> > lexical ENVIRONMENT (see static scoping) in which the function
> > was created."
> >
> >> If you create a closure, using a memoization technique as per the original
> >> post, and then call type() on that closure, Python reports <type 'function'>.

> >
> > that's because "closure" is an abstract concept.

>
> So are functions, modules, classes, instances, ints and strings.


they're hardly abstract; they all have distinct object implementations in the
Python runtime. there's an include file and a main implementation file for each
one of them, and you can access the type objects for each one of them from
Python code. any implementation of Python has to implement them. there's
no such thing for "closures"; they're the sum of a lot of different parts, all of
which can be implemented in many different ways.

> If what you say is true, then all functions are closures, and closure is
> just a synonym for function, and there is no difference between a function
> and a closure.


having something isn't the same thing as being something.

if you replace "function" with "function object", you get a bit closer to the
truth. e.g. in this snippet

def wrapper(a):
def func3():
return a
return func

"func" and "wrapper" are functions, but if you call wrapper twice, you get
two different function objects, each with it's own distinct closure.

(to see this in action, take the snippet I posted in an earlier post and
add

f = wrapper(10); decode(f)
f = wrapper(10); decode(f)

to the end. this prints:

--------------------------------------------------------------------
func3:
9 0 LOAD_DEREF 0 (a)
3 RETURN_VALUE
4 LOAD_CONST 0 (None)
7 RETURN_VALUE
co_freevars = ('a',)
func_closure = (<cell at 0x0083EE50: int object at 0x00798250>,)

--------------------------------------------------------------------
func3:
9 0 LOAD_DEREF 0 (a)
3 RETURN_VALUE
4 LOAD_CONST 0 (None)
7 RETURN_VALUE
co_freevars = ('a',)
func_closure = (<cell at 0x0083EDB0: int object at 0x0079A960>,)

)

same function, same code, different closures.

(note that in CPython, functions only exist on the syntax level; when
you writing something that you think is a function, the CPython com-
piler will turn this into a code object and translate "def" to bytecode;
the actual function object isn't created until you execute those byte-
codes)

> At a practical, Python level, there is a difference between a function
> before and after it gets made into a closure using, e.g. the original
> poster's memoization technique. In Python at least, it is not true that a
> function and a function-turned-into-closure is the same thing.


function or function object? all instances of the latter have a global en-
vironment (func_globals), some also have default arguments (func_defaults),
some also have references to nested scopes (func_freevars). all of them
have all these attributes; that they're set to None when empty is just an
memory optimization.

> >>> spam1.func_closure is None

> True
> >>> spam2.func_closure

> (<cell at 0xf70a2734: function object at 0xf6e0a144>,)
> >>> hex(id(spam1))

> '0xf6e0a144'
>
> In one case, the "raw" function has None stored in its func_closure
> attribute. In the other, it has a tuple containing at least one object of
> type cell. That cell object appears to contain a reference back to the
> original "raw" function.


func_closure is the name of an attribute. it contains information about
a certain part of a function object's closure (free variables in an existing
outer scope), but, despite its name, it's not *the* closure.

> So no, there is no "abstract" difference between a closure and a raw
> function, but there is a practical difference.


only if you're obsessed with CPython implementation details.

</F>



 
Reply With Quote
 
Diez B. Roggisch
Guest
Posts: n/a
 
      10-09-2005
> Clearly there is no DISTINCT closure object. If there were, I wouldn't
> need to ask how one can tell them apart, because type() would just report
> that one was a function and one was a closure. I don't have a problem with
> that. But read on...
>
>
>>function objects always con-
>>tain all the information they need about their context, and the sum of that is
>>what forms the closure.

>
>
> If what you say is true, then all functions are closures, and closure is
> just a synonym for function, and there is no difference between a function
> and a closure. Then why bother to create the term? Clearly, whoever
> invented the term did so to distinguish the two.


You seem to still not getting to it: they aren't supposed to be
distinguished - a function is a function. But it _can_ have a closure,
which (shortly spoken) enriches it's local variables lookup. Speaking in
terms of "is a" could be seen as some inheritance relation. Which
functions and functions with a closure _could_ have if one wanted to.
But that is true for other OO-concepts two. See below.

> At a practical, Python level, there is a difference between a function
> before and after it gets made into a closure using, e.g. the original
> poster's memoization technique. In Python at least, it is not true that a
> function and a function-turned-into-closure is the same thing.
>
> See, for example, this:-



<snip>

Nope. You mix things here. spam1 is a function. spam2 is actually bound
to the function-object created by do_nothing - a function that has a
closure because it is nested in closefy, and accesses variables from its
outer lexical scope - thus i needs a closure. Part of do_nothing's
closure is spam1 (or, better to say, the function reference that has
been originally bound to spam1 - spam1 is just a name). Important is
that the closure gets created at runtime, so each returned instance of
do_nothing has its own. But the original spam1 implementation is
untouched contrary to what you state above. It still is teh same
thing, and has no closure.

Consider this:

class Foo(object):
pass

f = Foo()
g = Foo()
g.bar = "whatever"

f and g are both Foo - but one has a different attribute set. And if Foo
had some semantics that did something different based on bar, would you
also say they're supposed have two different classes? You could in fact
do that - create one class for each incarnation of possible state. But
then the concept of classes and objects gets somewhat blurred, as each
instance would have its own class. The same is true for functions.

> So no, there is no "abstract" difference between a closure and a raw
> function, but there is a practical difference. Now perhaps in some other
> languages there is no practical difference either, but (as far as I can
> see) Python is not that language.
>
> Am I close?


Closer

Regards,

Diez
 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      10-09-2005
On Sun, 09 Oct 2005 13:43:38 +0200, Fredrik Lundh wrote:

>> If I wanted to inspect the value of cache, where would I find it?

>
> nowhere. at least no officially; see "Rebinding names in enclosing
> scopes" in the design document for a discussion:
>
> http://www.python.org/peps/pep-0227.html


That's interesting to read, and does clarify a few things that weren't
straight in my head, but I notice that the author, Jeremy Hylton,
distinguishes between Python creating function definitions that are
closures from those that aren't:

"Each def or lambda expression that is executed will create a closure if
the body of the function or any contained function has free variables."

Presumably that means that if there are no free variables, no closure
is created.


>> That assumes that cache can be seen in this way. If it can't, is this a
>> way to create "really private" variables in Python?

>
> no, because Python doesn't prevent you from digging into the
> internals:
>
> http://aspn.activestate.com/ASPN/Coo.../Recipe/440515


That's blackest voodoo, and as the warning says, could cause Python to
sigfault if you get the opcodes wrong.

I think that this is close enough to "really private" to satisfy most
people. Maybe even Paul Rubin *wink*


--
Steven.

 
Reply With Quote
 
Fredrik Lundh
Guest
Posts: n/a
 
      10-09-2005
Steven D'Aprano wrote:

> "Each def or lambda expression that is executed will create a closure if
> the body of the function or any contained function has free variables."
>
> Presumably that means that if there are no free variables, no closure
> is created.


you're quoting selectively; the word "closure" isn't used anywhere
in the design document except for a section that talks about "flat
closures", which is an implementation approach used in CPython
(as the sentence before the one you quoted explains).

I don't think it's necessarily a good idea to treat a reference to an
algorithm and an attribute on an object in a specific implementation
as the definition of an computer term...

</F>



 
Reply With Quote
 
Ron Adam
Guest
Posts: n/a
 
      10-09-2005
Fredrik Lundh wrote:
> Ron Adam wrote:
>
>
>>In effect, 'cache' and 'fn' are replaced by the objects they reference
>>before the cached_result function is returned.

>
>
> not really; accesses to "free variables" always go via special cell objects
> (rather than direct object references), and the compiler uses special byte
> codes for all accesses to such objects.
>
> this snippet illustrates the differences:
>
> def func1():
> return a
>
> def func2(a):
> return a
>
> def wrapper(a):
> def func3():
> return a
> return func3
>
> def decode(func):
> import dis
> print func.__name__ + ":"
> dis.dis(func)
> print "co_freevars =", func.func_code.co_freevars
> print "func_closure =", func.func_closure
> print
>
> decode(func1)
> decode(func2)
> decode(wrapper(10))
>
> if you run this on a recent version of Python, you get something like:
>
> func1:
> 2 0 LOAD_GLOBAL 0 (a)
> 3 RETURN_VALUE
> co_freevars = ()
> func_closure = None
>
> func2:
> 5 0 LOAD_FAST 0 (a)
> 3 RETURN_VALUE
> co_freevars = ()
> func_closure = None
>
> func3:
> 9 0 LOAD_DEREF 0 (a)
> 3 RETURN_VALUE
> co_freevars = ('a',)
> func_closure = (<cell at 0x0092C970: int object at 0x008A537C>,)
>
> note the use of LOAD_DEREF for the variable access.
>
> Other opcodes include STORE_DEREF (that updates a local variable
> through a cell, used inside the "defining" scope), and MAKE_CLOSURE
> (which sets up the function object for functions that actually uses
> nested scopes; this is where the func_closure attribute is initialized).
>
> (now, if I weren't limited to plain text, I could have drawn a nice little
> diagram for you, but that's an entirely different thread...)
>
> </F>


Thanks Fred, That's something useful to know when working with
decorators I think. And it should answer Stevens question too.

Abstract terminology is great once you already know the things and
concepts it refers to. But if you don't, finding where the abstract
connects to reality is sometimes not easy. There's nothing better than
a good example to illistrate the concept.

Cheers,
Ron

 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      10-09-2005
On Sun, 09 Oct 2005 18:00:13 +0200, Fredrik Lundh wrote:

> Steven D'Aprano wrote:
>
>> "Each def or lambda expression that is executed will create a closure if
>> the body of the function or any contained function has free variables."
>>
>> Presumably that means that if there are no free variables, no closure
>> is created.

>
> you're quoting selectively; the word "closure" isn't used anywhere
> in the design document except for a section that talks about "flat
> closures", which is an implementation approach used in CPython
> (as the sentence before the one you quoted explains).


Given that when I asked about closures, you pointed me at that document
for further information, I thought maybe I could take it as authoritative.

> I don't think it's necessarily a good idea to treat a reference to an
> algorithm and an attribute on an object in a specific implementation as
> the definition of an computer term...


Are you suggesting that Jeremy Hylton got it wrong? That's fine if he did,
but it would help if you said so rather than beating around the bush.

Is it correct to say that Python creates closures *ever*?

Is it correct to say that Python creates closures when a def or lambda
statement is executed?

Is it correct to say that Python *always* creates a closure whenever a def
or lambda is executed? Or is it only for *certain* defs/lambdas?

Is it correct to talk about functions *having* closures, or that they
*are* closures, or that the function is *part of* a closure?

If all functions are closures, what does it mean to say that some
languages (e.g. Java, C++, C#) don't have closures but merely simulate
them? Are there any languages that don't have closures?


Thanks for your patience, I'm sure I'll get there eventually.




--
Steven.

 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      10-09-2005
On Sun, 09 Oct 2005 17:55:16 +0200, Diez B. Roggisch wrote:

>> If what you say is true, then all functions are closures, and closure is
>> just a synonym for function, and there is no difference between a function
>> and a closure. Then why bother to create the term? Clearly, whoever
>> invented the term did so to distinguish the two.

>
> You seem to still not getting to it: they aren't supposed to be
> distinguished - a function is a function. But it _can_ have a closure,
> which (shortly spoken) enriches it's local variables lookup.


[penny drops] Now we're getting somewhere... a closure is something
_added_ to a function. So we should talk about functions-without-closures
and functions-with-closures.

> Speaking in
> terms of "is a" could be seen as some inheritance relation.


[penny rises again] Dammit, just when I thought I got it. So a closure is
like a subclass of function? Functions are a higher level classification,
like mammals. There are mammals that are primates (functions that _are_
closures) and mammals that aren't (functions that _aren't_ closures).


> Which
> functions and functions with a closure _could_ have if one wanted to.
> But that is true for other OO-concepts two. See below.
>
>> At a practical, Python level, there is a difference between a function
>> before and after it gets made into a closure using, e.g. the original
>> poster's memoization technique. In Python at least, it is not true that a
>> function and a function-turned-into-closure is the same thing.
>>
>> See, for example, this:-

>
>
> <snip>
>
> Nope. You mix things here. spam1 is a function.


Yes.

> spam2 is actually bound
> to the function-object created by do_nothing -


Yes.

> a function that has a closure because it is nested in closefy, and
> accesses variables from its outer lexical scope - thus i needs a
> closure.


Right! So closures are something which functions _have_ -- which is why
Python functions have an attribute called func_closure, which is empty in
functions without closures, and contain a tuple of cells in those with
them.

Now somebody is going to tell me this is an implementation detail and it
isn't true in the general language independent case...


> Part of do_nothing's closure is spam1 (or, better to say, the
> function reference that has been originally bound to spam1 - spam1 is
> just a name).


Yes, I can see that.

> Important is that the closure gets created at runtime, so
> each returned instance of do_nothing has its own. But the original spam1
> implementation is untouched contrary to what you state above. It still
> is teh same thing, and has no closure.
>
> Consider this:
>
> class Foo(object):
> pass
>
> f = Foo()
> g = Foo()
> g.bar = "whatever"
>
> f and g are both Foo


Now you're trying to trick me... they are both *instances* of Foo, not
Foo. Foo is a class, and f and g are not classes, let alone the same class.

> - but one has a different attribute set. And if Foo
> had some semantics that did something different based on bar, would you
> also say they're supposed have two different classes? You could in fact
> do that - create one class for each incarnation of possible state. But
> then the concept of classes and objects gets somewhat blurred, as each
> instance would have its own class. The same is true for functions.


Ah, but in languages like Pascal that can take functions as arguments, but
can't return functions as output, *all* functions have to be created
separately. To take your analogy and run with it, Pascal would be like a
language that didn't allow f and g to have different attributes unless
they belonged to different classes.

Python is not like that, which is why you can write a function to return
functions (a factory function?). If the output function needs access to
the namespace of the factory function, Python adds a closure to that
output function, giving it access to the objects in that namespace.


--
Steven.

 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      10-09-2005
On Sun, 09 Oct 2005 17:39:23 +0200, Fredrik Lundh wrote:

> only if you're obsessed with CPython implementation details.


No. I'm obsessed with finding out what closures are, since nobody seems to
have a good definition of them!

However, I have learnt some things: closures are something which functions
HAVE, not ARE. The func_closure attribute is just part of the
implementation of the closure. Some languages have closures and some
don't. And a closure is something that lets a function object access the
lexical scope that existed when the function object was created, e.g. the
namespace of the function which created it.

Am I getting there now?


--
Steven
who is very grateful for the time folks have taken to explain this.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
problem in running a basic code in python 3.3.0 that includes HTML file Satabdi Mukherjee Python 1 04-04-2013 07:48 PM
Why doesnt __getattr__ with decorator dont call __get_method in decorator glomde Python 5 03-29-2007 02:48 PM
ASp.NET caches assemblies in the GAC enrico sabbadin ASP .Net 2 06-03-2004 08:01 AM
ASP.net caches aspx Pages Thorsten Tarrach ASP .Net 8 04-08-2004 10:00 AM
IFrame caches content on local machine. Ken Dopierala Jr. ASP .Net 0 01-05-2004 07:27 PM



Advertisments