Velocity Reviews > closures and dynamic binding

# closures and dynamic binding

Guest
Posts: n/a

 09-28-2008
Hello all,

To me, this is a somewhat unintuitive behavior. I want to discuss the
parts of it I don't understand.

>>> f= [ None ]* 10
>>> for n in range( 10 ):

.... f[ n ]= lambda: n
....
>>> f[0]()

9
>>> f[1]()

9

I guess I can accept this part so far, though it took a little getting
used to. I'm writing some code and found the following workaround,
but I don't think it should give different results. Maybe I'm not
understanding some of the details of closures.

>>> f= [ None ]* 10
>>> for n in range( 10 ):

.... f[ n ]= (lambda n: ( lambda: n ) )( n )
....
>>> f[0]()

0
>>> f[1]()

1

Which is of course the desired effect. Why doesn't the second one
just look up what 'n' is when I call f[0], and return 9?

Marc 'BlackJack' Rintsch
Guest
Posts: n/a

 09-28-2008
On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi\" Brady wrote:

> To me, this is a somewhat unintuitive behavior. I want to discuss the
> parts of it I don't understand.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):

> ... f[ n ]= lambda: n
> ...
>>>> f[0]()

> 9
>>>> f[1]()

> 9

`n` is looked up at the time ``f[0]`` is called. At that time it is
bound to 9.

>>>> f= [ None ]* 10
>>>> for n in range( 10 ):

> ... f[ n ]= (lambda n: ( lambda: n ) )( n ) ...
>>>> f[0]()

> 0
>>>> f[1]()

> 1
>
> Which is of course the desired effect. Why doesn't the second one just
> look up what 'n' is when I call f[0], and return 9?

It *does* look up `n` at the time you call ``f[0]`` but this time it's
the local `n` of the outer ``lambda`` function and that is bound to
whatever the function was called with. At the time it was called the
global `n` was bound to 0. Maybe it get's more clear if you don't name
it `n`:

In [167]: f = [None] * 10

In [168]: for n in xrange(10):
.....: f[n] = (lambda x: lambda: x)(n)
.....:

In [169]: f[0]()
Out[169]: 0

Ciao,
Marc 'BlackJack' Rintsch

Guest
Posts: n/a

 09-28-2008
On Sep 28, 1:14*am, Marc 'BlackJack' Rintsch <(E-Mail Removed)> wrote:
> On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi\" Brady wrote:
> > To me, this is a somewhat unintuitive behavior. *I want to discuss the
> > parts of it I don't understand.

>
> >>>> f= [ None ]* 10
> >>>> for n in range( 10 ):

> > ... * * f[ n ]= lambda: n
> > ...
> >>>> f[0]()

> > 9
> >>>> f[1]()

> > 9

>
> `n` is looked up at the time ``f[0]`` is called. *At that time it is
> bound to 9.
>
> >>>> f= [ None ]* 10
> >>>> for n in range( 10 ):

> > ... * * f[ n ]= (lambda n: ( lambda: n ) )( n ) ...
> >>>> f[0]()

> > 0
> >>>> f[1]()

> > 1

>
> > Which is of course the desired effect. *Why doesn't the second one just
> > look up what 'n' is when I call f[0], and return 9?

>
> It *does* look up `n` at the time you call ``f[0]`` but this time it's
> the local `n` of the outer ``lambda`` function and that is bound to
> whatever the function was called with. *At the time it was called the
> global `n` was bound to 0. *Maybe it get's more clear if you don't name
> it `n`:
>
> In [167]: f = [None] * 10
>
> In [168]: for n in xrange(10):
> * *.....: * * f[n] = (lambda x: lambda: x)(n)
> * *.....:
>
> In [169]: f[0]()
> Out[169]: 0
>
> Ciao,
> * * * * Marc 'BlackJack' Rintsch

Hi Marc,

It's my understanding that 'n' gets a new value every time through the
loop. n= 0 on the first pass, n= 1 on the second pass, and so on. n=
9 by the end, and that's why `lambda: n` always returns 9. It queries
the variable 'n', and finds 9. (This got lengthy. I started thinking
aloud.)

In your version of the indirect example, it queries the variable 'x',
which it finds in a new distinct scope in each f element. In f[0], x=
0. In f[1], x= 1. There are 10 different 'x' variables throughout
the contents of f. The direct example does not do this allocation of
ten different 'x's.

It's sort of helping. I think I feel like the following is more like
what I'm looking for:

(Non-standard)
>>> f= [ None ]* 10
>>> for n in range( 10 ):

.... f[ n ]= n
....
>>> f[0]

9
>>> f[1]

9

because when you access f[0], it looks up the variable 'n'. Obviously
not.

(Unproduced)
>>> f= [ None ]* 10
>>> for n in range( 10 ):

.... f[ n ]= late( n ) #late/lazy
....
>>> f[0]

9
>>> f[1]

9

>>> f= [ None ]* 10
>>> for n in range( 10 ):

.... f[ n ]= early( n ) #early/eager
....
>>> f[0]

0
>>> f[1]

1

For the functions, I want a function that returns 'n', either late or
early.

(Unproduced)
>>> for n in range( 10 ):

.... f[ n ]= lambda: late( n )
>>> f[0]()

9
>>> for n in range( 10 ):

.... f[ n ]= lambda: early( n )
>>> f[0]()

0

I don't think you could pull this off. 'late' and 'early' succeed
with quotes around n, early('n') and late('n'), in the direct
assignments, but the functions aren't so lucky. 'n' has gone on to a
better world by the time 'early' gets any control.

This might have some success.

(Unproduced)
>>> for n in range( 10 ):

.... f[ n ]= late( lambda: n )
>>> f[0]()

9
>>> for n in range( 10 ):

.... f[ n ]= early( lambda: n )
>>> f[0]()

0

Though it's beyond my foresight to tell if it's feasible as stated, if
you need quotes, how much introspection you would need, etc. Plus,
'late' and 'early' were accepting quoted parameters earlier. How
would they look inside a function? Could a simple decorator provide
the service?

On a tangent about mutables, it's not that numbers, strings, and
tuples are 'immutable' per se, it's just that they don't have any
methods which mutate them (or assignable properties). Lists and
dictionaries do. It's up to the user whether a custom class does.

Terry Reedy
Guest
Posts: n/a

 09-28-2008
> Hello all,
>
> To me, this is a somewhat unintuitive behavior. I want to discuss the
> parts of it I don't understand.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):

> ... f[ n ]= lambda: n

This is equivalent to

for n in range(10):
def g(): return n
f[n] = g

which is equivalent to

def g(): return n
f = [g]*10
n = 9

>>>> f[0]()

> 9
>>>> f[1]()

> 9

which make this not so surprising as the original lambda version is to
some people.

> I guess I can accept this part so far, though it took a little getting
> used to. I'm writing some code and found the following workaround,
> but I don't think it should give different results. Maybe I'm not
> understanding some of the details of closures.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):

> ... f[ n ]= (lambda n: ( lambda: n ) )( n )

This is equivalent to

for n in range(10):
def g(n):
def h:
return n
return h
f[n] = g(n)

Now, to avoid the needless confusion of 'n's, g is equivalent to

def g(x):
def h:
return x
return h

(One could do the same change in the lambdas, too, of course).
so that g(n)() == n, with n stored in each closure h...

> ...
>>>> f[0]()

> 0
>>>> f[1]()

> 1

to be regurgitated when each is called.

Terry Jan Reedy

Steven D'Aprano
Guest
Posts: n/a

 09-28-2008
On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi\" Brady wrote:

> Hello all,
>
> To me, this is a somewhat unintuitive behavior. I want to discuss the
> parts of it I don't understand.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):

> ... f[ n ]= lambda: n
> ...
>>>> f[0]()

> 9
>>>> f[1]()

> 9
>
> I guess I can accept this part so far, though it took a little getting
> used to. I'm writing some code and found the following workaround, but
> I don't think it should give different results. Maybe I'm not
> understanding some of the details of closures.
>
>>>> f= [ None ]* 10
>>>> for n in range( 10 ):

> ... f[ n ]= (lambda n: ( lambda: n ) )( n )
> ...
>>>> f[0]()

> 0
>>>> f[1]()

> 1
>
> Which is of course the desired effect. Why doesn't the second one just
> look up what 'n' is when I call f[0], and return 9?

That's an awfully complicated solution. A much easier way to get the
result you are after is to give each function its own local copy of n:

f[n] = lambda n=n: n

As for why the complicated version works, it may be clearer if you expand
it from a one-liner:

# expand: f[ n ]= (lambda n: ( lambda: n ) )( n )

inner = lambda: n
outer = lambda n: inner
f[n] = outer(n)

outer(0) => inner with a local scope of n=0
outer(1) => inner with a local scope of n=1 etc.

Then, later, when you call inner() it grabs the local scope and returns
the number you expected.

--
Steven

Guest
Posts: n/a

 09-28-2008
On Sep 28, 2:52*am, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.au> wrote:
> On Sat, 27 Sep 2008 21:43:15 -0700, Aaron \"Castironpi\" Brady wrote:
> > Hello all,

>
> > To me, this is a somewhat unintuitive behavior. *I want to discuss the
> > parts of it I don't understand.

>
> >>>> f= [ None ]* 10
> >>>> for n in range( 10 ):

> > ... * * f[ n ]= lambda: n
> > ...
> >>>> f[0]()

> > 9
> >>>> f[1]()

> > 9

>
> > I guess I can accept this part so far, though it took a little getting
> > used to. *I'm writing some code and found the following workaround, but
> > I don't think it should give different results. *Maybe I'm not
> > understanding some of the details of closures.

>
> >>>> f= [ None ]* 10
> >>>> for n in range( 10 ):

> > ... * * f[ n ]= (lambda n: ( lambda: n ) )( n )
> > ...
> >>>> f[0]()

> > 0
> >>>> f[1]()

> > 1

>
> > Which is of course the desired effect. *Why doesn't the second one just
> > look up what 'n' is when I call f[0], and return 9?

>
> That's an awfully complicated solution. A much easier way to get the
> result you are after is to give each function its own local copy of n:
>
> f[n] = lambda n=n: n
>
> As for why the complicated version works, it may be clearer if you expand
> it from a one-liner:
>
> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
>
> inner = lambda: n
> outer = lambda n: inner
> f[n] = outer(n)
>
> outer(0) => inner with a local scope of n=0
> outer(1) => inner with a local scope of n=1 etc.
>
> Then, later, when you call inner() it grabs the local scope and returns
> the number you expected.
>
> --
> Steven

Steven,

I must have misunderstood. Here's my run of your code:

>>> inner = lambda: n
>>> outer = lambda n: inner
>>> outer(0)

<function <lambda> at 0x00A01170>
>>> a=outer(0)
>>> b=outer(1)
>>> a()

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <lambda>
NameError: global name 'n' is not defined

Why doesn't 'inner' know it's been used in two different scopes, and
look up 'n' based on the one it's in?

Terry Reedy
Guest
Posts: n/a

 09-28-2008
> On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-

>> As for why the complicated version works, it may be clearer if you expand
>> it from a one-liner:
>>
>> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
>>
>> inner = lambda: n
>> outer = lambda n: inner
>> f[n] = outer(n)
>>
>> outer(0) => inner with a local scope of n=0
>> outer(1) => inner with a local scope of n=1 etc.

For this to work, the 'expansion' has to be mental and not actual.
Which is to say, inner must be a text macro to be substituted back into
outer.

>> Then, later, when you call inner() it grabs the local scope and returns
>> the number you expected.

>
> I must have misunderstood. Here's my run of your code:

I cannot speak to what Steven meant, but

>>>> inner = lambda: n

when inner is actually compiled outside of outer, it is no longer a
closure over outer's 'n' and 'n' will be looked for in globals instead.

>>>> outer = lambda n: inner
>>>> outer(0)

> <function <lambda> at 0x00A01170>
>>>> a=outer(0)
>>>> b=outer(1)
>>>> a()

> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "<stdin>", line 1, in <lambda>
> NameError: global name 'n' is not defined
>
> Why doesn't 'inner' know it's been used in two different scopes, and
> look up 'n' based on the one it's in?

That would be dynamic rather than lexical scoping.

Steven D'Aprano
Guest
Posts: n/a

 09-28-2008
On Sun, 28 Sep 2008 17:47:44 -0400, Terry Reedy wrote:

>> On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-

>
>>> As for why the complicated version works, it may be clearer if you
>>> expand it from a one-liner:
>>>
>>> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )
>>>
>>> inner = lambda: n
>>> outer = lambda n: inner
>>> f[n] = outer(n)
>>>
>>> outer(0) => inner with a local scope of n=0 outer(1) => inner with a
>>> local scope of n=1 etc.

>
> For this to work, the 'expansion' has to be mental and not actual. Which
> is to say, inner must be a text macro to be substituted back into outer.

Er, yes, that's what I meant, sorry for not being more explicit. That's
why it wasn't a copy and paste of actual running code.

Or perhaps I just confused myself and was talking nonsense.

--
Steven

Guest
Posts: n/a

 09-28-2008
On Sep 28, 4:47*pm, Terry Reedy <(E-Mail Removed)> wrote:
> > On Sep 28, 2:52 am, Steven D'Aprano <st...@REMOVE-THIS-
> >> As for why the complicated version works, it may be clearer if you expand
> >> it from a one-liner:

>
> >> # expand: f[ n ]= (lambda n: ( lambda: n ) )( n )

>
> >> inner = lambda: n
> >> outer = lambda n: inner
> >> f[n] = outer(n)

>
> >> outer(0) => inner with a local scope of n=0
> >> outer(1) => inner with a local scope of n=1 etc.

>
> For this to work, the 'expansion' has to be mental and not actual.
> Which is to say, inner must be a text macro to be substituted back into
> outer.
>
> >> Then, later, when you call inner() it grabs the local scope and returns
> >> the number you expected.

>
> > I must have misunderstood. *Here's my run of your code:

>
> I cannot speak to what Steven meant, but
>
> >>>> inner = lambda: n

>
> when inner is actually compiled outside of outer, it is no longer a
> closure over outer's 'n' and 'n' will be looked for in globals instead.
>
> >>>> outer = lambda n: inner
> >>>> outer(0)

> > <function <lambda> at 0x00A01170>
> >>>> a=outer(0)
> >>>> b=outer(1)
> >>>> a()

> > Traceback (most recent call last):
> > * File "<stdin>", line 1, in <module>
> > * File "<stdin>", line 1, in <lambda>
> > NameError: global name 'n' is not defined

>
> > Why doesn't 'inner' know it's been used in two different scopes, and
> > look up 'n' based on the one it's in?

>
> That would be dynamic rather than lexical scoping.

I couldn't find how those apply on the wikipedia website. It says:
"dynamic scoping can be dangerous and almost no modern languages use
it", but it sounded like that was what closures use. Or maybe it was
what 'inner' in Steven's example would use. I'm confused.

Actually, I'll pick this apart a little bit. See above when I
suggested 'late' and 'early' functions which control (or simulate)
different bindings. I get the idea that 'late' bound functions would
use a dangerous "dynamic scope", but I could be wrong; that's just my
impression.

> >> inner = lambda: n
> >> outer = lambda n: inner
> >> f[n] = outer(n)

>
> >> outer(0) => inner with a local scope of n=0
> >> outer(1) => inner with a local scope of n=1 etc.

If you defined these as:

inner= late( lambda: n )
outer= lambda n: inner

You could get the right results. It's not even clear you need
quotes. Perhaps 'late' could carry the definition of 'n' with it when
it's returned from 'outer'.

In my proposal, it makes a copy of the "localest" namespace, at least
all the variables used below it, then returns its argument in an
original closure.

Terry Reedy
Guest
Posts: n/a

 09-29-2008
> On Sep 28, 4:47 pm, Terry Reedy <(E-Mail Removed)> wrote:

>>>>>> inner = lambda: n

>> when inner is actually compiled outside of outer, it is no longer a
>> closure over outer's 'n' and 'n' will be looked for in globals instead.
>>
>>>>>> outer = lambda n: inner
>>>>>> outer(0)
>>> <function <lambda> at 0x00A01170>
>>>>>> a=outer(0)
>>>>>> b=outer(1)
>>>>>> a()
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in <module>
>>> File "<stdin>", line 1, in <lambda>
>>> NameError: global name 'n' is not defined
>>> Why doesn't 'inner' know it's been used in two different scopes, and
>>> look up 'n' based on the one it's in?

>> That would be dynamic rather than lexical scoping.

>
> I couldn't find how those apply on the wikipedia website. It says:
> "dynamic scoping can be dangerous and almost no modern languages use
> it", but it sounded like that was what closures use. Or maybe it was
> what 'inner' in Steven's example would use. I'm confused.

As I understand it, partly from postings here years ago...

Lexical: The namespace scope of 'n' in inner is determined by where
inner is located in the code -- where is is compiled. This is Python
(and nearly all modern languages). Even without closures, the global
scope of a function is the module it is defined in.

Dynamic: The namespace scope of 'n' in inner, how it is looked up, is
determined by where inner is called from. This is what you seemed to be
suggesting -- look up 'n' based on the scope it is *used* in.

Even without closures, dynamic scoping would be if the global scope of a
function for each call were the module it is called in.

tjr