Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Dangerous behavior of list(generator)

Reply
Thread Tools

Re: Dangerous behavior of list(generator)

 
 
exarkun@twistedmatrix.com
Guest
Posts: n/a
 
      12-14-2009
On 02:58 pm, http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
>(E-Mail Removed) wrote:
>>On 08:45 am, (E-Mail Removed) wrote:
>>>Tom Machinski wrote:
>>>>In most cases, `list(generator)` works as expected. Thus,
>>>>`list(<generator expression>)` is generally equivalent to
>>>>`[<generator
>>>>expression>]`.
>>>>
>>>>Here's a minimal case where this equivalence breaks, causing a
>>>>serious
>>>>and hard-to-detect bug in a program:
>>>>
>>>> >>> def sit(): raise StopIteration()
>>>
>>>StopIteration is intended to be used only within the .__next__ method
>>>of iterators. The devs know that other 'off-label' use results in the
>>>inconsistency you noted, but their and my view is 'don't do that'.

>>
>>Which is unfortunate, because it's not that hard to get StopIteration
>>without explicitly raising it yourself and this behavior makes it
>>difficult to debug such situations.
>>
>>What's with this view, exactly? Is it just that it's hard to
>>implement
>>the more desirable behavior?

>
>I'm not exactly sure what you're asking for.
>
>The StopIteration exception originated as part of the for-loop
>protocol. Later on it was generalized to apply to generators
>as well.
>
>The reason for using an exception is simple: raising and catching
>exceptions is fast at C level and since the machinery for
>communicating exceptions up the call stack was already there
>(and doesn't interfere with the regular return values), this
>was a convenient method to let the upper call levels know
>that an iteration has ended (e.g. a for-loop 4 levels up the
>stack).
>
>I'm not sure whether that answers your question, but it's the
>reason for things being as they are


I'm asking about why the behavior of a StopIteration exception being
handled from the `expression` of a generator expression to mean "stop
the loop" is accepted by "the devs" as acceptable. To continue your
comparison to for loops, it's as if a loop like this:

for a in b:
c

actually meant this:

for a in b:
try:
c
except StopIteration:
break

Note, I know *why* the implementation leads to this behavior. I'm
asking why "the devs" *accept* this.

Jean-Paul
 
Reply With Quote
 
 
 
 
Mel
Guest
Posts: n/a
 
      12-14-2009
(E-Mail Removed) wrote:
[ ... ]
it's as if a loop like this:
>
> for a in b:
> c
>
> actually meant this:
>
> for a in b:
> try:
> c
> except StopIteration:
> break
>
> Note, I know *why* the implementation leads to this behavior. I'm
> asking why "the devs" *accept* this.


It's part of the price Python pays for letting people get their hands on the
controls. Consider also:

Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class dict2(dict):

.... def __getitem__ (self, key):
.... if key == 'fatal':
.... raise KeyError
....
>>> d = dict2()
>>> d['fatal'] = 'Hello, world!'
>>> print d['fatal']

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 4, in __getitem__
KeyError
>>>



"KeyError when we just put the item into the dict?"
"Yep."


Mel.

>
> Jean-Paul



 
Reply With Quote
 
 
 
 
Carl Banks
Guest
Posts: n/a
 
      12-14-2009
On Dec 14, 7:21*am, (E-Mail Removed) wrote:
> Note, I know *why* the implementation leads to this behavior. *I'm
> asking why "the devs" *accept* this.


As noted, the problem isn't with generators but with iteration
protocol. The devs "allowed" this because it was a necessary evil for
correct functionality. As the system is set up it can't discriminate
between a legitimate and a spurrious StopIteration. (Which is why
Steven and others called you out for suggesting that the compiler has
to read your mind.)

However, as far as I'm concerned there is no reasonable argument that
this behavior is good. So how, hypothetically, would one go about
fixing it, short of ripping out and replacing the existing machinery?

The first argument is that StopIteration has no place within a
generator expression. Therefore a generator expression (but not a
generator function) could deliberately catch StopIteration and raise a
different exception.

I don't like it, though: who says that StopIteration has no place
within a generator expression? Currently it's possible to do
something like this to terminate a genexp early, and I won't the one
saying you shouldn't do it.

def stop(): raise StopIteration

list(x or stop() for x in stream)

(Though personally I'd bite the bullet and write it as a generator
function).

What else? The way I see it, when you throw StopIteration you are
trying to stop a specific generator. Therefore StopIteration should
(somehow) contain a reference to the generator that it's being applied
to. Perhaps it can be obtained by crawling ths stack, which shouldn't
be a significant penalty (it'd be only called once per iteration, and
most of the time it'd be only one or two frames up). The looping
logic within Python should check whether the reference matches the
object it's iterating over; if not it raises a LeakyLoopException or
something like that.

I haven't thought this out though, I'm just kind of throwing this out
there. Any issues?


But to answer your question, I think "simple is better than complex"
rules the day. Right now StopIteration stops an iteration, simple as
that. Any fix would add complexity.


Carl Banks
 
Reply With Quote
 
Michele Simionato
Guest
Posts: n/a
 
      12-16-2009
On Dec 14, 11:05*pm, Carl Banks <(E-Mail Removed)> wrote:
> But to answer your question, I think "simple is better than complex"
> rules the day. *Right now StopIteration stops an iteration, simple as
> that. *Any fix would add complexity.


+1
 
Reply With Quote
 
Martin v. Loewis
Guest
Posts: n/a
 
      01-02-2010
> I'm asking about why the behavior of a StopIteration exception being
> handled from the `expression` of a generator expression to mean "stop
> the loop" is accepted by "the devs" as acceptable.


I may be late to this discussion, but the answer is "most definitely
yes". *Any* exception leads to termination of the iterator, and
StopIteration is no different:

py> def stop(e):
.... def f():
.... raise e
.... return f
....
py> g = (f() for f in (lambda:1,stop(StopIteration),lambda:2))
py> g.next
<method-wrapper 'next' of generator object at 0xb7960fac>
py> g.next()
1
py> g.next()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <genexpr>
File "<stdin>", line 3, in f
StopIteration
py> g.next()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
py> g = (f() for f in (lambda:1,stop(ValueError),lambda:2))
py> g.next()
1
py> g.next()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <genexpr>
File "<stdin>", line 3, in f
ValueError
py> g.next()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration

Regards,
Martin
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Dangerous behavior of list(generator) Gabriel Genellina Python 14 01-02-2010 08:17 PM
Re: Dangerous behavior of list(generator) exarkun@twistedmatrix.com Python 3 12-15-2009 06:26 AM
Re: Dangerous behavior of list(generator) exarkun@twistedmatrix.com Python 10 12-14-2009 10:35 AM
Dangerous behavior of list(generator) Tom Machinski Python 0 12-13-2009 12:15 AM
undefined behavior or not undefined behavior? That is the question Mantorok Redgormor C Programming 70 02-17-2004 02:46 PM



Advertisments