Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Dangerous behavior of list(generator)

Reply
Thread Tools

Re: Dangerous behavior of list(generator)

 
 
exarkun@twistedmatrix.com
Guest
Posts: n/a
 
      12-13-2009
On 08:45 am, http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
>Tom Machinski wrote:
>>In most cases, `list(generator)` works as expected. Thus,
>>`list(<generator expression>)` is generally equivalent to `[<generator
>>expression>]`.
>>
>>Here's a minimal case where this equivalence breaks, causing a serious
>>and hard-to-detect bug in a program:
>>
>> >>> def sit(): raise StopIteration()

>
>StopIteration is intended to be used only within the .__next__ method
>of iterators. The devs know that other 'off-label' use results in the
>inconsistency you noted, but their and my view is 'don't do that'.


Which is unfortunate, because it's not that hard to get StopIteration
without explicitly raising it yourself and this behavior makes it
difficult to debug such situations.

What's with this view, exactly? Is it just that it's hard to implement
the more desirable behavior?

Jean-Paul
 
Reply With Quote
 
 
 
 
Steven D'Aprano
Guest
Posts: n/a
 
      12-13-2009
On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote:

>>StopIteration is intended to be used only within the .__next__ method of
>>iterators. The devs know that other 'off-label' use results in the
>>inconsistency you noted, but their and my view is 'don't do that'.

>
> Which is unfortunate, because it's not that hard to get StopIteration
> without explicitly raising it yourself and this behavior makes it
> difficult to debug such situations.


I can't think of any way to get StopIteration without explicitly raising
it yourself. It's not like built-ins or common data structures routinely
raise StopIteration. I don't think I've *ever* seen a StopIteration that
I didn't raise myself.


> What's with this view, exactly? Is it just that it's hard to implement
> the more desirable behavior?


What is that "more desirable behaviour"? That StopIteration is used to
signal that Python should stop iterating except when you want it to be
ignored? Unfortunately, yes, it's quite hard to implement "do what the
caller actually wants, not what he asked for" behaviour -- and even if it
were possible, it goes against the grain of the Zen of Python.

If you've ever had to debug faulty "Do What I Mean" software, you'd see
this as a good thing.



--
Steven
 
Reply With Quote
 
 
 
 
exarkun@twistedmatrix.com
Guest
Posts: n/a
 
      12-13-2009
On 08:18 pm, (E-Mail Removed) wrote:
>On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote:
>>>StopIteration is intended to be used only within the .__next__ method
>>>of
>>>iterators. The devs know that other 'off-label' use results in the
>>>inconsistency you noted, but their and my view is 'don't do that'.

>>
>>Which is unfortunate, because it's not that hard to get StopIteration
>>without explicitly raising it yourself and this behavior makes it
>>difficult to debug such situations.

>
>I can't think of any way to get StopIteration without explicitly
>raising
>it yourself. It's not like built-ins or common data structures
>routinely
>raise StopIteration. I don't think I've *ever* seen a StopIteration
>that
>I didn't raise myself.


Call next on an iterator. For example: iter(()).next()
>
>>What's with this view, exactly? Is it just that it's hard to
>>implement
>>the more desirable behavior?

>
>What is that "more desirable behaviour"? That StopIteration is used to
>signal that Python should stop iterating except when you want it to be
>ignored? Unfortunately, yes, it's quite hard to implement "do what the
>caller actually wants, not what he asked for" behaviour -- and even if
>it
>were possible, it goes against the grain of the Zen of Python.
>
>If you've ever had to debug faulty "Do What I Mean" software, you'd see
>this as a good thing.


I have plenty of experience developing and debugging software, Steven.
Your argument is specious, as it presupposes that only two possibilities
exist: the current behavior of some kind of magical faerie land
behavior.

I'm surprised to hear you say that the magical faerie land behavior
isn't desirable either, though. I'd love a tool that did what I wanted,
not what I asked. The only serious argument against this, I think, is
that it is beyond our current ability to create (and so anyone claiming
to be able to do it is probably mistaken).

You chopped out all the sections of this thread which discussed the more
desirable behavior. You can go back and read them in earlier messages
if you need to be reminded. I'm not talking about anything beyond
what's already been raised.

I'm pretty sure I know the answer to my question, though - it's hard to
implement, so it's not implemented.

Jean-Paul
 
Reply With Quote
 
Lie Ryan
Guest
Posts: n/a
 
      12-14-2009
On 12/14/2009 9:45 AM, (E-Mail Removed) wrote:
> On 08:18 pm, (E-Mail Removed) wrote:
>> On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote:
>>>> StopIteration is intended to be used only within the .__next__
>>>> method of
>>>> iterators. The devs know that other 'off-label' use results in the
>>>> inconsistency you noted, but their and my view is 'don't do that'.
>>>
>>> Which is unfortunate, because it's not that hard to get StopIteration
>>> without explicitly raising it yourself and this behavior makes it
>>> difficult to debug such situations.

>>
>> I can't think of any way to get StopIteration without explicitly raising
>> it yourself. It's not like built-ins or common data structures routinely
>> raise StopIteration. I don't think I've *ever* seen a StopIteration that
>> I didn't raise myself.

>
> Call next on an iterator. For example: iter(()).next()


..next() is not meant to be called directly, that's why it's renamed to
..__next__() in python 3. Just like .__add__() is never meant to be
called directly since you'll then have to handle NotImplemented.

If you really need to call .__next__() you will call next() builtin
function instead which has a second argument to return a sentinel value
instead of StopIteration. IMNSHO next()'s sentinel should always be
specified except if you're implementing __next__() or if the sequence is
an infinite iterator.

>>> What's with this view, exactly? Is it just that it's hard to implement
>>> the more desirable behavior?

>>
>> What is that "more desirable behaviour"? That StopIteration is used to
>> signal that Python should stop iterating except when you want it to be
>> ignored? Unfortunately, yes, it's quite hard to implement "do what the
>> caller actually wants, not what he asked for" behaviour -- and even if it
>> were possible, it goes against the grain of the Zen of Python.
>>
>> If you've ever had to debug faulty "Do What I Mean" software, you'd see
>> this as a good thing.

>
> I have plenty of experience developing and debugging software, Steven.
> Your argument is specious, as it presupposes that only two possibilities
> exist: the current behavior of some kind of magical faerie land behavior.
>
> I'm surprised to hear you say that the magical faerie land behavior
> isn't desirable either, though. I'd love a tool that did what I wanted,
> not what I asked. The only serious argument against this, I think, is
> that it is beyond our current ability to create (and so anyone claiming
> to be able to do it is probably mistaken).


In your world, this is what happens:
>>> list = [a, b, c]
>>> # print list
>>> print list

["a", "b", "c"]
>>> # make a copy of list
>>> alist = list(llst) # oops a mistype
>>> alist = alist - "]" + ", "d"]"
>>> print alist

["a", "b", "c", "d"]
>>> alist[:6] + "i", + alist[6:]
>>> print alist

["a", "i", "b", "c", "d"]
>>> print alist
>>> # hearing the sound of my deskjet printer...
>>> C:\fikle.text.write(alist)
>>> print open("C:\file.txt").read()

<h1>a</h1>
<ul>
<li>i</li>
<li>b</li>
<li>c d</li>
>>> # great, exactly what I needed


> You chopped out all the sections of this thread which discussed the more
> desirable behavior. You can go back and read them in earlier messages if
> you need to be reminded. I'm not talking about anything beyond what's
> already been raised.
>
> I'm pretty sure I know the answer to my question, though - it's hard to
> implement, so it's not implemented.
>
> Jean-Paul


 
Reply With Quote
 
exarkun@twistedmatrix.com
Guest
Posts: n/a
 
      12-14-2009
On 02:50 am, (E-Mail Removed) wrote:
>On 12/14/2009 9:45 AM, (E-Mail Removed) wrote:
>>On 08:18 pm, (E-Mail Removed) wrote:
>>>On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote:
>>>>>StopIteration is intended to be used only within the .__next__
>>>>>method of
>>>>>iterators. The devs know that other 'off-label' use results in the
>>>>>inconsistency you noted, but their and my view is 'don't do that'.
>>>>
>>>>Which is unfortunate, because it's not that hard to get
>>>>StopIteration
>>>>without explicitly raising it yourself and this behavior makes it
>>>>difficult to debug such situations.
>>>
>>>I can't think of any way to get StopIteration without explicitly
>>>raising
>>>it yourself. It's not like built-ins or common data structures
>>>routinely
>>>raise StopIteration. I don't think I've *ever* seen a StopIteration
>>>that
>>>I didn't raise myself.

>>
>>Call next on an iterator. For example: iter(()).next()

>
>.next() is not meant to be called directly


Doesn't matter. Sometimes it makes sense to call it directly. And I
was just giving an example of a way to get StopIteration raised without
doing it yourself - which is what Steve said he couldn't think of.
>>
>>I'm surprised to hear you say that the magical faerie land behavior
>>isn't desirable either, though. I'd love a tool that did what I
>>wanted,
>>not what I asked. The only serious argument against this, I think, is
>>that it is beyond our current ability to create (and so anyone
>>claiming
>>to be able to do it is probably mistaken).

>
>In your world, this is what happens:
> >>> list = [a, b, c]
> >>> # print list
> >>> print list

>["a", "b", "c"]
> >>> # make a copy of list
> >>> alist = list(llst) # oops a mistype
> >>> alist = alist - "]" + ", "d"]"
> >>> print alist

>["a", "b", "c", "d"]
> >>> alist[:6] + "i", + alist[6:]
> >>> print alist

>["a", "i", "b", "c", "d"]
> >>> print alist
> >>> # hearing the sound of my deskjet printer...
> >>> C:\fikle.text.write(alist)
> >>> print open("C:\file.txt").read()

><h1>a</h1>
><ul>
><li>i</li>
><li>b</li>
><li>c d</li>
> >>> # great, exactly what I needed


I don't understand the point of this code listing, sorry. I suspect you
didn't completely understand the magical faerie land I was describing -
where all your programs would work, no matter what mistakes you made
while writing them.

Jean-Paul
 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      12-14-2009
On Sun, 13 Dec 2009 22:45:58 +0000, exarkun wrote:

> On 08:18 pm, (E-Mail Removed) wrote:
>>On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote:
>>>>StopIteration is intended to be used only within the .__next__ method
>>>>of
>>>>iterators. The devs know that other 'off-label' use results in the
>>>>inconsistency you noted, but their and my view is 'don't do that'.
>>>
>>>Which is unfortunate, because it's not that hard to get StopIteration
>>>without explicitly raising it yourself and this behavior makes it
>>>difficult to debug such situations.

>>
>>I can't think of any way to get StopIteration without explicitly raising
>>it yourself. It's not like built-ins or common data structures routinely
>>raise StopIteration. I don't think I've *ever* seen a StopIteration that
>>I didn't raise myself.

>
> Call next on an iterator. For example: iter(()).next()


Or in more recent versions of Python, next(iter(())).

Good example. But next() is a special case, and since next() is
documented as raising StopIteration if you call it and it raises
StopIteration, you have raised it yourself. Just not explicitly.



>>>What's with this view, exactly? Is it just that it's hard to implement
>>>the more desirable behavior?

>>
>>What is that "more desirable behaviour"? That StopIteration is used to
>>signal that Python should stop iterating except when you want it to be
>>ignored? Unfortunately, yes, it's quite hard to implement "do what the
>>caller actually wants, not what he asked for" behaviour -- and even if
>>it were possible, it goes against the grain of the Zen of Python.
>>
>>If you've ever had to debug faulty "Do What I Mean" software, you'd see
>>this as a good thing.

>
> I have plenty of experience developing and debugging software, Steven.
> Your argument is specious, as it presupposes that only two possibilities
> exist: the current behavior of some kind of magical faerie land
> behavior.
>
> I'm surprised to hear you say that the magical faerie land behavior
> isn't desirable either, though. I'd love a tool that did what I wanted,
> not what I asked. The only serious argument against this, I think, is
> that it is beyond our current ability to create (and so anyone claiming
> to be able to do it is probably mistaken).


I'd argue that tools that do what you want rather than what you ask for
are not just currently impossible, but always will be -- no matter how
good the state of the art of artificial intelligent mind-reading software
becomes.



> You chopped out all the sections of this thread which discussed the more
> desirable behavior. You can go back and read them in earlier messages
> if you need to be reminded. I'm not talking about anything beyond
> what's already been raised.


I'm glad for you. But would you mind explaining for those of us aren't
mind-readers what YOU consider the "more desirable behaviour"?

If you're talking the list constructor and list comprehensions treating
StopIteration the same, then I don't think it is at all self-evident that
the current behaviour is a bad thing, nor that the only reason for it is
that to do otherwise would be hard.

(I don't think it would be hard to have list comps swallow a
StopIteration.)


> I'm pretty sure I know the answer to my question, though - it's hard to
> implement, so it's not implemented.
>
> Jean-Paul



--
Steven
 
Reply With Quote
 
exarkun@twistedmatrix.com
Guest
Posts: n/a
 
      12-14-2009
On 04:11 am, (E-Mail Removed) wrote:
>On Sun, 13 Dec 2009 22:45:58 +0000, exarkun wrote:
>>On 08:18 pm, (E-Mail Removed) wrote:
>>>On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote:
>>>>>StopIteration is intended to be used only within the .__next__
>>>>>method
>>>>>of
>>>>>iterators. The devs know that other 'off-label' use results in the
>>>>>inconsistency you noted, but their and my view is 'don't do that'.
>>>>
>>>>Which is unfortunate, because it's not that hard to get
>>>>StopIteration
>>>>without explicitly raising it yourself and this behavior makes it
>>>>difficult to debug such situations.
>>>
>>>I can't think of any way to get StopIteration without explicitly
>>>raising
>>>it yourself. It's not like built-ins or common data structures
>>>routinely
>>>raise StopIteration. I don't think I've *ever* seen a StopIteration
>>>that
>>>I didn't raise myself.

>>
>>Call next on an iterator. For example: iter(()).next()

>
>Or in more recent versions of Python, next(iter(())).
>
>Good example. But next() is a special case, and since next() is
>documented as raising StopIteration if you call it and it raises
>StopIteration, you have raised it yourself. Just not explicitly.


But if you mistakenly don't catch it, and you're trying to debug your
code to find this mistake, you probably won't be aided in this pursuit
by the exception-swallowing behavior of generator expressions.
>
>>>>What's with this view, exactly? Is it just that it's hard to
>>>>implement
>>>>the more desirable behavior?
>>>
>>>What is that "more desirable behaviour"? That StopIteration is used
>>>to
>>>signal that Python should stop iterating except when you want it to
>>>be
>>>ignored? Unfortunately, yes, it's quite hard to implement "do what
>>>the
>>>caller actually wants, not what he asked for" behaviour -- and even
>>>if
>>>it were possible, it goes against the grain of the Zen of Python.
>>>
>>>If you've ever had to debug faulty "Do What I Mean" software, you'd
>>>see
>>>this as a good thing.

>>
>>I have plenty of experience developing and debugging software, Steven.
>>Your argument is specious, as it presupposes that only two
>>possibilities
>>exist: the current behavior of some kind of magical faerie land
>>behavior.
>>
>>I'm surprised to hear you say that the magical faerie land behavior
>>isn't desirable either, though. I'd love a tool that did what I
>>wanted,
>>not what I asked. The only serious argument against this, I think, is
>>that it is beyond our current ability to create (and so anyone
>>claiming
>>to be able to do it is probably mistaken).

>
>I'd argue that tools that do what you want rather than what you ask for
>are not just currently impossible, but always will be -- no matter how
>good the state of the art of artificial intelligent mind-reading
>software
>becomes.


That may be true. I won't try to make any predictions about the
arbitrarily distant future, though.
>>You chopped out all the sections of this thread which discussed the
>>more
>>desirable behavior. You can go back and read them in earlier messages
>>if you need to be reminded. I'm not talking about anything beyond
>>what's already been raised.

>
>I'm glad for you. But would you mind explaining for those of us aren't
>mind-readers what YOU consider the "more desirable behaviour"?


The behavior of list comprehensions is pretty good. The behavior of
constructing a list out of a generator expression isn't as good. The
behavior which is more desirable is for a StopIteration raised out of
the `expression` part of a `generator_expression` to not be treated
identically to the way a StopIteration raised out of the `genexpr_for`
part is. This could provide behavior roughly equivalent to the behavior
of a list comprehension.
>
>If you're talking the list constructor and list comprehensions treating
>StopIteration the same, then I don't think it is at all self-evident
>that
>the current behaviour is a bad thing, nor that the only reason for it
>is
>that to do otherwise would be hard.


I don't expect it to be self-evident. I wasn't even trying to convince
anyone that it's desirable (although I did claim it, so I won't fault
anyone for making counter-arguments). The only thing I asked was what
the motivation for the current behavior is. If the motivation is that
it is self-evident that the current behavior is the best possible
behavior, then someone just needs to say that and my question is
answered.

Jean-Paul
 
Reply With Quote
 
Terry Reedy
Guest
Posts: n/a
 
      12-14-2009
On 12/13/2009 10:29 PM, (E-Mail Removed) wrote:

> Doesn't matter. Sometimes it makes sense to call it directly.


It only makes sense to call next (or .__next__) when you are prepared to
explicitly catch StopIteration within a try..except construct.
You did not catch it, so it stopped execution.

Let me repeat: StopIteration is intended only for stopping iteration.
Outside that use, it is a normal exception with no special meaning.

Terry Jan Reedy

 
Reply With Quote
 
Terry Reedy
Guest
Posts: n/a
 
      12-14-2009
On 12/13/2009 11:33 PM, (E-Mail Removed) wrote:
> But if you mistakenly don't catch it, and you're trying to debug your
> code to find this mistake, you probably won't be aided in this pursuit
> by the exception-swallowing behavior of generator expressions.


As I remember, it was the call to list that swalled the exception, not
the generator expression. List() takes an iterable as arg and stopping
on StopIteration is what it does and how it knows to stop and return the
new list.

> The behavior of list comprehensions is pretty good. The behavior of
> constructing a list out of a generator expression isn't as good.


I think you are confused. A generator expression is a shorthand for a
def statement that defines a generator function followed by a call to
the generator function to get a generator followed by deletion of the
function. When you call list() to make a list, it constructs the list
from the generator, not from the expression itself. List has no idea
that you used a generator expression or even that it was passed a
generator. Leaving error checks out, it operates something like

def list(it):
res = []
it = iter(it)
for item in it: # stops whenever it raises StopIteration
res.append(item)
return res

> The
> behavior which is more desirable is for a StopIteration raised out of
> the `expression` part of a `generator_expression` to not be treated
> identically to the way a StopIteration raised out of the `genexpr_for`
> part is.


It is not. StopIteration in for part stops the for loop in the
generator. StopIteration in the expression part stops the loop in the
list() call (sooner than it would have been otherwise). When the
generator raises StopIteration, list() has no idea what statement within
the body raised it. It MUST stop.

> This could provide behavior roughly equivalent to the behavior
> of a list comprehension.


Impossible. The only serious option for consistency is to special case
list comps to also trap StopIteration raised in the expression part, but
the devs decided not to do this as doing do is arguably a bug.

Terry Jan Reedy

 
Reply With Quote
 
Lie Ryan
Guest
Posts: n/a
 
      12-14-2009
On 12/14/09, (E-Mail Removed) <(E-Mail Removed)> wrote:
> On 02:50 am, (E-Mail Removed) wrote:
>>On 12/14/2009 9:45 AM, (E-Mail Removed) wrote:
>>>On 08:18 pm, (E-Mail Removed) wrote:
>>>>On Sun, 13 Dec 2009 14:35:21 +0000, exarkun wrote:
>>>>>>StopIteration is intended to be used only within the .__next__
>>>>>>method of
>>>>>>iterators. The devs know that other 'off-label' use results in the
>>>>>>inconsistency you noted, but their and my view is 'don't do that'.
>>>>>
>>>>>Which is unfortunate, because it's not that hard to get
>>>>>StopIteration
>>>>>without explicitly raising it yourself and this behavior makes it
>>>>>difficult to debug such situations.
>>>>
>>>>I can't think of any way to get StopIteration without explicitly
>>>>raising
>>>>it yourself. It's not like built-ins or common data structures
>>>>routinely
>>>>raise StopIteration. I don't think I've *ever* seen a StopIteration
>>>>that
>>>>I didn't raise myself.
>>>
>>>Call next on an iterator. For example: iter(()).next()

>>
>>.next() is not meant to be called directly

>
> Doesn't matter. Sometimes it makes sense to call it directly. And I
> was just giving an example of a way to get StopIteration raised without
> doing it yourself - which is what Steve said he couldn't think of.
>>>
>>>I'm surprised to hear you say that the magical faerie land behavior
>>>isn't desirable either, though. I'd love a tool that did what I
>>>wanted,
>>>not what I asked. The only serious argument against this, I think, is
>>>that it is beyond our current ability to create (and so anyone
>>>claiming
>>>to be able to do it is probably mistaken).

>>
>>In your world, this is what happens:
>> >>> list = [a, b, c]
>> >>> # print list
>> >>> print list

>>["a", "b", "c"]
>> >>> # make a copy of list
>> >>> alist = list(llst) # oops a mistype
>> >>> alist = alist - "]" + ", "d"]"
>> >>> print alist

>>["a", "b", "c", "d"]
>> >>> alist[:6] + "i", + alist[6:]
>> >>> print alist

>>["a", "i", "b", "c", "d"]
>> >>> print alist
>> >>> # hearing the sound of my deskjet printer...
>> >>> C:\fikle.text.write(alist)
>> >>> print open("C:\file.txt").read()

>><h1>a</h1>
>><ul>
>><li>i</li>
>><li>b</li>
>><li>c d</li>
>> >>> # great, exactly what I needed

>
> I don't understand the point of this code listing, sorry. I suspect you
> didn't completely understand the magical faerie land I was describing -
> where all your programs would work, no matter what mistakes you made
> while writing them.


Exactly, that's what's happening. It just works. It knows that when I
said alist[:6] + "i", + alist[6:] ; I want to insert "i" between the
sixth character of the textual representation of the list. It knows to
find the correct variable when I made a typo. It correctly guess that
I want to print to a paper instead of to screen. It knows that when I
wrote to C:\path.write(), it knows I wanted a HTML output in that
specific format. It just works (TM), whatever mistakes I made. That's
what you wanted, right?
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Dangerous behavior of list(generator) exarkun@twistedmatrix.com Python 4 01-02-2010 08:27 PM
Re: Dangerous behavior of list(generator) Gabriel Genellina Python 14 01-02-2010 08:17 PM
Re: Dangerous behavior of list(generator) exarkun@twistedmatrix.com Python 3 12-15-2009 06:26 AM
Dangerous behavior of list(generator) Tom Machinski Python 0 12-13-2009 12:15 AM
undefined behavior or not undefined behavior? That is the question Mantorok Redgormor C Programming 70 02-17-2004 02:46 PM



Advertisments