Velocity Reviews > Something is rotten in Denmark...

# Something is rotten in Denmark...

harrismh777
Guest
Posts: n/a

 05-31-2011
>>> fs=[]
>>> fs = [(lambda n: i + n) for i in range(10)]
>>> [fs[i](1) for i in range(10)]

[10, 10, 10, 10, 10, 10, 10, 10, 10, 10] <=== not good

( that was a big surprise! . . . )
( let's try it another way . . . )

>>> fs =[]
>>> def g(i): return (lambda n: i + n)
>>> fs = [g(i) for i in range(10)]
>>> [fs[i](1) for i in range(10)]

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

(aaah, that's better . . . )
(hmmm, let's try another . . . )

>>> fs =[]
>>> for i in range(10):

fs.append(lambda n, i=i: i + n)
>>> [fs[i](1) for i in range(10)]

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

(aaaah, that works too . . . )
(... and another... )

>>> fs=[]
>>> fs = [(lambda n, i=i: i + n) for i in range(10)]
>>> [fs[i](1) for i in range(10)]

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

(great! . . . )
(now then, what's wrong with this picture ?)

lambda? closure? scope? bug?

What is going on with the binding in the first construct... this
seems to reduce the usefulness of lambda to a considerable extent?

kind regards,
m harris

Chris Rebert
Guest
Posts: n/a

 05-31-2011
On Mon, May 30, 2011 at 11:48 PM, harrismh777 <(E-Mail Removed)> wrote:
>>>> fs=[]
>>>> fs = [(lambda n: i + n) for i in range(10)]
>>>> [fs[i](1) for i in range(10)]

>
> [10, 10, 10, 10, 10, 10, 10, 10, 10, 10] Â* Â* Â* Â* <=== not good
>
> Â* Â*( that was a big surprise! . . . )

<snip>
> Â* Â* lambda? Â*closure? Â*scope? Â* bug?
>
> Â* Â* What is going on with the binding in the first construct...this seems
> to reduce the usefulness of lambda to a considerable extent?

http://stackoverflow.com/questions/2...ures-in-python

Cheers,
Chris

Ian Kelly
Guest
Posts: n/a

 05-31-2011
On Tue, May 31, 2011 at 12:48 AM, harrismh777 <(E-Mail Removed)> wrote:
> * * What is going on with the binding in the first construct... this seems
> to reduce the usefulness of lambda to a considerable extent?

I don't see why; as you've shown there are a couple of simple ways to
avoid this problem. The trick is just to recognize that when you have
a closure around a variable, and that variable needs to change, but
the value in the closure needs to be constant, then what you really
need are two separate variables -- the cell variable needs to be
promoted to a local. How you accomplish that is not terribly
important.

One other technique that is sometimes preferable is functools.partial, e.g.:

fs = [functools.partial(operator.add, i) for i in range(10)]

Tangentially, I'd like to point out that this line:

[fs[i](1) for i in range(10)]

is more naturally written as:

[f(1) for f in fs]

Jussi Piitulainen
Guest
Posts: n/a

 05-31-2011
harrismh777 writes:

> >>> fs=[]
> >>> fs = [(lambda n: i + n) for i in range(10)]
> >>> [fs[i](1) for i in range(10)]

> [10, 10, 10, 10, 10, 10, 10, 10, 10, 10] <=== not good
>
> ( that was a big surprise! . . . )
> ( let's try it another way . . . )

The ten functions share the same i. The list comprehension changes the
value of that i. At the time when the functions are called, the value
is 9.

A different list comprehension mechanism could create a fresh i for
each element instead of changing the value of one i. Then each of the
functions would have a private i which would have the value it had at
the time of the creation of the closure. That is not the Python
mechanism.

The same sharing-an-i thing happens here:

>>> fs = []
>>> for i in range(4):

.... fs.append(lambda n : i + n)
....
>>> fs[0](0)

3

And the different private-j thing happens here:

>>> gs = []
>>> for i in range(4):

.... gs.append((lambda j : lambda n : j + n)(i))
....
>>> gs[0](0)

0

You used the lambda itself to introduce its private i in your other
examples, in (lambda n, i=i : i + n). In its i=i, the i to the left is
a different i - will be a fresh i every time the function is called, I
think - while the i to the right gets resolved to the value of the i
that the list comprehension is stepping at the time when the closure
is created.

> What is going on with the binding in the first
> construct... this seems to reduce the usefulness of lambda to a
> considerable extent?

The lambda is doing its lambda thing exactly. The list comprehension
just updates the one i in whatever you call it that each of the ten
closures remember, and they all observe the updates, so to say.

It's a bit subtle. Using different names might help, like I used j.

Thomas Rachel
Guest
Posts: n/a

 05-31-2011
Am 31.05.2011 12:08 schrieb Jussi Piitulainen:

> The same sharing-an-i thing happens here:
>
>>>> fs = []
>>>> for i in range(4):

> ... fs.append(lambda n : i + n)
> ...
>>>> fs[0](0)

> 3
>
> And the different private-j thing happens here:
>
>>>> gs = []
>>>> for i in range(4):

> ... gs.append((lambda j : lambda n : j + n)(i))
> ...
>>>> gs[0](0)

> 0

There is a simpler way: with

>>>> fs = []
>>>> for i in range(4):

> ... fs.append(lambda n, i=i: i + n)
> ...

you give each lambda a different default argument.

>>>> fs[0](0)

> 0

Thomas

Jussi Piitulainen
Guest
Posts: n/a

 05-31-2011
Thomas Rachel writes:
> Am 31.05.2011 12:08 schrieb Jussi Piitulainen:
>
> > The same sharing-an-i thing happens here:
> >
> >>>> fs = []
> >>>> for i in range(4):

> > ... fs.append(lambda n : i + n)
> > ...
> >>>> fs[0](0)

> > 3
> >
> > And the different private-j thing happens here:
> >
> >>>> gs = []
> >>>> for i in range(4):

> > ... gs.append((lambda j : lambda n : j + n)(i))
> > ...
> >>>> gs[0](0)

> > 0

>
> There is a simpler way: with
>
> >>>> fs = []
> >>>> for i in range(4):

> > ... fs.append(lambda n, i=i: i + n)
> > ...

>
> you give each lambda a different default argument.
>
> >>>> fs[0](0)

> > 0

I know, and harrismh777 knows. I find it an unnecessary distraction
when explaining why the different closures in the initial example
behave identically, but I did discuss it at the end of my post.

Terry Reedy
Guest
Posts: n/a

 05-31-2011
On 5/31/2011 2:48 AM, harrismh777 wrote:
>>>> fs=[]

Irrelevant here since you immediately rebind 'fs'.

>>>> fs = [(lambda n: i + n) for i in range(10)]
>>>> [fs[i](1) for i in range(10)]

Same as [f(1) for f in fs]

> [10, 10, 10, 10, 10, 10, 10, 10, 10, 10] <=== not good
>
> ( that was a big surprise! . . . )

You have been hypnotizeed by lambda. (lambda n: i+n) is a *constant
expression*, so you get 10 'equal' functions. To see this better

fs = [(lambda n: i + n) for i in range(10)]
from dis import dis
for f in fs: dis(f)

7 RETURN_VALUE

7 RETURN_VALUE
....

All have the same bytecode and all retrieve the same last value of i in
the nonlocal listcomp scope when you call them *after* the listcomp
scope has otherwise disappeared.

fs = []
for i in range(10):
fs.append(lambda n: i + n)
print([f(1) for f in fs])

which is equivalent (except for naming the functions) to

fs = []
for i in range(10):
def f(n): return i + n
fs.append(f)
print([f(1) for f in fs])

Does [10, 10, 10, 10, 10, 10, 10, 10, 10, 10] still surprise?
Because the def is a constant expression, we can move it out of the
loop, and get the equivalent (except for identity)

def f(n): return i + n
fs = []
for i in range(10):
fs.append(f)
print([f(1) for f in fs])

This in turn is equivalent to

def f(n): return i + n
fs = []
for _ in range(10):
fs.append(f)
i=9
print([f(1) for f in fs])

which in turn is equivalent in output to

def f(n): return i + n
i = 9
print([f(1) for _ in range(10)])

Note that:

def f(n): return i+n # or f = lambda n: i+n
fs = [f for i in range(10)]
print([f(1) for f in fs])

works in 2.7, with the same output, but not in 3.2 because in 3.x, i is
local to the list comp and the later call raises unbound global error.

> ( let's try it another way . . . )

All these other ways create 10 *different* (unequal) functions that are
different because they have captured 10 different values of i when
defined instead of deferring lookup of i to when they are called.

def g(i): return (lambda n: i + n)
fs = [g(i) for i in range(10)]
print([f.__closure__[0].cell_contents for f in fs])

fs = [(lambda n, i=i: i + n) for i in range(10)]
print([f.__defaults__[0] for f in fs])

# CPython 3.2 dependent code !!!

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

--
Terry Jan Reedy

harrismh777
Guest
Posts: n/a

 05-31-2011
Terry Reedy wrote:
> You have been hypnotizeed by lambda. (lambda n: i+n) is a *constant
> expression*, so you get 10 'equal' functions.

'hypnotized' indeed! ... ok, so let me see if I get this... the lambda
defers lookup|bind of its references until such time as the lambda is
'called' and not at the time (as I thought) that the anonymous
function(s) are returned?

If I'm understanding that correctly, then that means lambda is working
as designed, and that there are very subtle nuances to be aware of. In
my little case

(lambda n: i + n)

... if the i goes out of scope before the anonymous function gets
called then we have a problem... or if i as a reference is mutable or
refers to a different object before the anonymous function is called
then we have a problem?

What I am discovering is that 'yes' I can use lambda syntactically
where I might not be able to code a def statement; however, if I do use
it (as in a list comprehension) then I may get unexpected results if any
of my lambda's references go out of scope or are mutable...(?)

Question:

What are the ramifications of making the lookup|binding happen at
the time the anonymous function is 'returned' vs 'called'? Has anyone
suggested this? Is this PEP-able....? Are there side-effects in the
other direction?

PS Thanks Chris, Ian, Jussi, Thomas, Terry... I appreciate your
teaching and patience !

kind regards,
m harris

Martin Manns
Guest
Posts: n/a

 05-31-2011
On Tue, 31 May 2011 01:48:05 -0500
harrismh777 <(E-Mail Removed)> wrote:

> >>> fs=[]
> >>> fs = [(lambda n: i + n) for i in range(10)]
> >>> [fs[i](1) for i in range(10)]

> [10, 10, 10, 10, 10, 10, 10, 10, 10, 10] <=== not good
>
> ( that was a big surprise! . . . )
> ( let's try it another way . . . )

After being confused I figured out it is a 3.x example:

----

\$ python
Python 2.6.6 (r266:84292, Apr 20 2011, 11:58:30)
[GCC 4.5.2] on linux2
>>> fs=[]
>>> fs = [(lambda n: i + n) for i in range(10)]
>>> [fs[i](1) for i in range(10)]

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

----

\$ python3.1
Python 3.1.3 (r313:86834, Nov 28 2010, 11:28:10)
[GCC 4.4.5] on linux2
>>> fs=[]
>>> fs = [(lambda n: i + n) for i in range(10)]
>>> [fs[i](1) for i in range(10)]

[10, 10, 10, 10, 10, 10, 10, 10, 10, 10]

----

Is this fixed automatically by 2to3?

Martin

Ian Kelly
Guest
Posts: n/a

 05-31-2011
On Tue, May 31, 2011 at 3:14 PM, Martin Manns <(E-Mail Removed)> wrote:
> \$ python
> Python 2.6.6 (r266:84292, Apr 20 2011, 11:58:30)
> [GCC 4.5.2] on linux2
>>>> fs=[]
>>>> fs = [(lambda n: i + n) for i in range(10)]
>>>> [fs[i](1) for i in range(10)]

> [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

This works by accident.

>>> [fs[i](1) for i in range(10)]

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> [fs[0](1) for i in range(10)]

[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
>>> [f(1) for f in fs]

[10, 10, 10, 10, 10, 10, 10, 10, 10, 10]

The i variable is part of the global scope, and as you iterate over
range(10) again it coincidentally takes on the same values as in the
original list comprehension. You don't see this in Python 3 because
the scope of i is limited to the list comprehension, not global.

Cheers,
Ian