Velocity Reviews > StopIteration in the if clause of a generator expression

# StopIteration in the if clause of a generator expression

Peter Otten
Guest
Posts: n/a

 04-01-2005
To confuse a newbies and old hands alike, Bengt Richter wrote:

> Need something more straightforward, e.g., a wrapped one-liner:
>
> >>> def guess(n=3): print ("You're right!", 'No more tries for
> >>> you!!!')[n-1 in

> ... (x for x in xrange(n) for t in [raw_input('Guess my name:
> ')=='Ben']
> ... if not t or iter([]).next())]
> ...
> >>> guess()

To make it a bit clearer, a StopIteration raised in a generator expression
silently terminates that generator:

>>> def stop(): raise StopIteration

....
>>> list(i for i in range(10) if i < 5 or stop())

[0, 1, 2, 3, 4]

In a list comprehension, on the other hand, it is propagated:

>>> [i for i in range(10) if i < 5 or stop()]

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 1, in stop
StopIteration

Is that an intentional difference?

Peter

Carl Banks
Guest
Posts: n/a

 04-01-2005
Peter Otten wrote:
> To confuse a newbies and old hands alike, Bengt Richter wrote:
>
> > Need something more straightforward, e.g., a wrapped one-liner:
> >
> > >>> def guess(n=3): print ("You're right!", 'No more tries for
> > >>> you!!!')[n-1 in

> > ... (x for x in xrange(n) for t in [raw_input('Guess my name:
> > ')=='Ben']
> > ... if not t or iter([]).next())]
> > ...
> > >>> guess()

>
> To make it a bit clearer, a StopIteration raised in a generator

expression
> silently terminates that generator:
>
> >>> def stop(): raise StopIteration

> ...
> >>> list(i for i in range(10) if i < 5 or stop())

> [0, 1, 2, 3, 4]
>
> In a list comprehension, on the other hand, it is propagated:
>
> >>> [i for i in range(10) if i < 5 or stop()]

> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "<stdin>", line 1, in stop
> StopIteration
>
> Is that an intentional difference?

Very interesting. I'm not sure if the designers even considered this
particular subtlety. Why it happens is pretty plain. In the generator
expression case, the generator expression does propogate the
StopIteration, but list() traps it. List comprehensions are internally
treated as a for-loop (kind of), which doesn't trap StopIteration.
Maybe it should.

The list comprehension [ x for x in y ] is currently treated as
equivalent to the following, with byte-code optimizations:

.. _ = []
.. for x in y:
.. _.append(x)

Perhaps it ought to be equivalent to:

.. _ = []
.. try:
.. for x in y:
.. _.append(x)
.. except StopIteration:
.. pass

However, I would guess the Python gods wouldn't approve of this use of
StopIteration, and so would make no sacrifices to get it.
Nevertheless, it seems likely to be how a list comprehension would
behave in Python 3.0, so maybe we should do it.

--
CARL BANKS

Raymond Hettinger
Guest
Posts: n/a

 04-01-2005
[Peter Otten]
> a StopIteration raised in a generator expression
> silently terminates that generator:
>
> >>> def stop(): raise StopIteration

> ...
> >>> list(i for i in range(10) if i < 5 or stop())

> [0, 1, 2, 3, 4]
>
> In a list comprehension, on the other hand, it is propagated:
>
> >>> [i for i in range(10) if i < 5 or stop()]

> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "<stdin>", line 1, in stop
> StopIteration
>
> Is that an intentional difference?

I would call it an unfortunate assymmetry -- one the never comes up unless
you're up to no good

In a way, both behave identically. They both raise StopIteration. In the case
of the generator expression, that StopIteration is intercepted by the enclosing
list() call. That becomes obvious if you write a pure python equivalent for
list:

def lyst(s):
it = iter(s)
result = []
try:
while 1:
result.append(it.next())
except StopIteration: # guess who trapped StopIter
return result

Raymond Hettinger

Bengt Richter
Guest
Posts: n/a

 04-01-2005
On Fri, 01 Apr 2005 16:34:32 GMT, "Raymond Hettinger" <(E-Mail Removed)> wrote:

>[Peter Otten]
>> a StopIteration raised in a generator expression
>> silently terminates that generator:
>>
>> >>> def stop(): raise StopIteration

>> ...
>> >>> list(i for i in range(10) if i < 5 or stop())

>> [0, 1, 2, 3, 4]
>>
>> In a list comprehension, on the other hand, it is propagated:
>>
>> >>> [i for i in range(10) if i < 5 or stop()]

>> Traceback (most recent call last):
>> File "<stdin>", line 1, in ?
>> File "<stdin>", line 1, in stop
>> StopIteration
>>
>> Is that an intentional difference?

>
>I would call it an unfortunate assymmetry -- one the never comes up unless
>you're up to no good

>
>In a way, both behave identically. They both raise StopIteration. In the case
>of the generator expression, that StopIteration is intercepted by the enclosing
>list() call. That becomes obvious if you write a pure python equivalent for
>list:
>
> def lyst(s):
> it = iter(s)
> result = []
> try:
> while 1:
> result.append(it.next())
> except StopIteration: # guess who trapped StopIter
> return result
>
>

I assumed that all standard sequence consumers (including list, of course) would intercept
the StopIteration of a sequence given them in the form of a generator expression, so your
lyst example would have an analogue for other sequence consumers as well, right?
I.e., there's not a hidden list(genex) in those others I would hope

E.g., "in" in my toy exposed more clearly, using Peter's stop:

>>> def show(x): print x,; return x

...
>>> def stop(): raise StopIteration

...
>>> 2 in (x for x in xrange(5) if show(x)<4 or stop())

0 1 2
True
>>> 7 in (x for x in xrange(5) if show(x)<4 or stop())

0 1 2 3 4
False

BTW I notice that this also nicely shortcuts when the 2 is found.

Regards,
Bengt Richter

Raymond Hettinger
Guest
Posts: n/a

 04-02-2005
> I assumed that all standard sequence consumers (including list, of course)
would intercept
> the StopIteration of a sequence given them in the form of a generator

expression, so your
> lyst example would have an analogue for other sequence consumers as well,

right?
> I.e., there's not a hidden list(genex) in those others I would hope

Right.

> E.g., "in" in my toy exposed more clearly, using Peter's stop:
>
> >>> def show(x): print x,; return x

> ...
> >>> def stop(): raise StopIteration

> ...
> >>> 2 in (x for x in xrange(5) if show(x)<4 or stop())

> 0 1 2
> True
> >>> 7 in (x for x in xrange(5) if show(x)<4 or stop())

> 0 1 2 3 4
> False
>
> BTW I notice that this also nicely shortcuts when the 2 is found.

That's a fact.

Raymond

jfj
Guest
Posts: n/a

 04-02-2005
Peter Otten wrote:
> To confuse a newbies and old hands alike, Bengt Richter wrote:

got me for one

>
> To make it a bit clearer, a StopIteration raised in a generator expression
> silently terminates that generator:

*any* exception raised from a generator, terminates the generator

jfj

Peter Otten
Guest
Posts: n/a

 04-03-2005
Raymond Hettinger wrote:

(quoting Bengt)
>> I assumed that all standard sequence consumers (including list, of
>> course) would intercept the StopIteration of a sequence given them in the
>> form of a generator expression, so your lyst example would have an
>> analogue for other sequence consumers as well, right?
>> I.e., there's not a hidden list(genex) in those others I would hope

>
> Right.

I see I followed the historical evolvement and saw generator expressions as
a lazy listcomp rather than a cool new way to write a generator. That
turned out to be the road to confusion.

Thanks Carl, thanks Raymond for setting me straight.

> I would call it an unfortunate assymmetry -- one the never comes up unless
> you're up to no good

Do you see any chance that list comprehensions will be redefined as an
alternative spelling for list(<generator expression>)?

Peter

Peter Otten
Guest
Posts: n/a

 04-03-2005
jfj wrote:

>> To make it a bit clearer, a StopIteration raised in a generator
>> expression silently terminates that generator:

>
> *any* exception raised from a generator, terminates the generator

Yeah, but StopIteration is the only expected exception and therefore the
only one that client code (nearly) always knows to deal with:

>>> def choke(): raise ValueError

....
>>> list(i for i in range(10) if i < 3 or choke())

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 1, in <generator expression>
File "<stdin>", line 1, in choke
ValueError
>>> [i for i in range(10) if i < 3 or choke()]

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "<stdin>", line 1, in choke
ValueError

Here you can *not* tell apart list(genexp) and listcomp.

(Of course, as has since been pointed out, the StopIteration is actually
caught in the list constructor, so nothing magic to the example in my
initial post)

Peter

Raymond Hettinger
Guest
Posts: n/a

 04-03-2005
[Peter Otten]
> Do you see any chance that list comprehensions will be redefined as an
> alternative spelling for list(<generator expression>)?

Not likely. It is possible that the latter spelling would make it possible for
Py3.0. eliminate list comps entirely. However, they are very popular and
practical, so my bet is that they will live on.

The more likely change is that in Py3.0 list comps will no longer expose the
loop variable outside the loop.

Raymond Hettinger

Steven Bethard
Guest
Posts: n/a

 04-03-2005
Raymond Hettinger wrote:
> [Peter Otten]
>
>>Do you see any chance that list comprehensions will be redefined as an
>>alternative spelling for list(<generator expression>)?

>
> Not likely. It is possible that the latter spelling would make it possible for
> Py3.0. eliminate list comps entirely. However, they are very popular and
> practical, so my bet is that they will live on.

I suspect you're right, but I certainly wouldn't complain if list comps
disappeared. TOOWTDI and all, and I often find myself alternating
between the two when I can't decide which one seems more Pythonic.
(These days I generally write a listcomp, but I wouldn't put any money

STeVe