Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > subtle side effect of generator/generator expression

Reply
Thread Tools

subtle side effect of generator/generator expression

 
 
bonono@gmail.com
Guest
Posts: n/a
 
      10-16-2005
Hi,

I initially thought that generator/generator expression is cool(sort of
like the lazy evaluation in Haskell) until I notice this side effect.

>>>a=(x for x in range(2))
>>>list(a)

[1,2]
>>>list(a)

[]

Would this make generator/generator expression's usage pretty limited ?
As when the program/system goes beyond a single module, this behaviour
can cause subtle bugs ?

 
Reply With Quote
 
 
 
 
Fredrik Lundh
Guest
Posts: n/a
 
      10-16-2005
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

> I initially thought that generator/generator expression is cool (sort of
> like the lazy evaluation in Haskell) until I notice this side effect.
>
> >>>a=(x for x in range(2))
> >>>list(a)

> [1,2]
> >>>list(a)

> []
>
> Would this make generator/generator expression's usage pretty limited ?


nope.

> As when the program/system goes beyond a single module, this behaviour
> can cause subtle bugs ?


sure, in the same way as

>>> f = open(filename)
>>> f.read()

'hello world\n'
>>> f.read() # oops!

''

causes subtle bugs (that is, almost never)

</F>



 
Reply With Quote
 
 
 
 
bonono@gmail.com
Guest
Posts: n/a
 
      10-16-2005
That is exactly what I meant, in fact. These IO thing are expected to
have side effects so they are not subtle. Generator on the other hand,
is sort of "clever iteratables".

Now that I notice that, Of course I can be sure I would be careful. But
what about the following situation :

I import some function from some third party module which said, "oh, my
function returns a iteratable". Then I can only list(o) if I want to
pass it around or I consume it in only one and only one place. turning
it into a list completely negate what generator is intended
for(generate as much as you need) and using it in only one and only one
place(like your fread example) would IMO means its usage is pretty
limited.

Beside, I can do a fseek to rewind a file handle but not a generator
object.

Fredrik Lundh wrote:
>
> > As when the program/system goes beyond a single module, this behaviour
> > can cause subtle bugs ?

>
> sure, in the same way as
>
> >>> f = open(filename)
> >>> f.read()

> 'hello world\n'
> >>> f.read() # oops!

> ''
>
> causes subtle bugs (that is, almost never)
>
> </F>


 
Reply With Quote
 
Diez B. Roggisch
Guest
Posts: n/a
 
      10-16-2005
(E-Mail Removed) wrote:
> That is exactly what I meant, in fact. These IO thing are expected to
> have side effects so they are not subtle. Generator on the other hand,
> is sort of "clever iteratables".
>
> Now that I notice that, Of course I can be sure I would be careful. But
> what about the following situation :
>
> I import some function from some third party module which said, "oh, my
> function returns a iteratable". Then I can only list(o) if I want to
> pass it around or I consume it in only one and only one place. turning
> it into a list completely negate what generator is intended
> for(generate as much as you need) and using it in only one and only one
> place(like your fread example) would IMO means its usage is pretty
> limited.
>
> Beside, I can do a fseek to rewind a file handle but not a generator
> object.


Its a question of protocol. A iterator only guarantes that you can fetch
items from it until it is possibly exhausted. In the same way, a stream
may only guarantee you that you can read from it. If you need more, you
have tgo use a different protocol. Lists are one that allows you to
randomly access it. Files allow to seek, in addition to stream semantics.

And given that python is an imperative language with side-effects, you
don't know what sideeffects a generator can cause - which makes a
"rewind" or copy-semantics to make it work several time impossible (in
the general case, that is). So it has to focus on what it _can_ guarantee.

Haskell and other FP languages canwork differently here, as sideeffects
are made excplicit using e.g. monads or other means - which allows for
re-use of a generator-like construct. But there is no way to make this
work in pythons paradigm.

Diez
 
Reply With Quote
 
Fredrik Lundh
Guest
Posts: n/a
 
      10-16-2005
Diez B. Roggisch wrote:

> Files allow to seek, in addition to stream semantics.


Some files. Not all files support seek operations. Some only support
forward seek.

</F>



 
Reply With Quote
 
Simon Percivall
Guest
Posts: n/a
 
      10-16-2005
If you find that you want to iterate over an iterable multiple times,
have a look at the solution that the tee() function in the itertools
module provides (http://docs.python.org/lib/itertools-functions.html).
(Have a look at the rest of the itertools module as well, for that
matter.)

 
Reply With Quote
 
bonono@gmail.com
Guest
Posts: n/a
 
      10-16-2005
True. That is why I have now reverted back to use list whenever
possible. As while list can also be modified(say in a multi-thread
situation), at least if I don't do the update(coding policy, practice
or whatever), they are sort of "guaranteed".

I would only use generator as IO monad in Haskell, i.e. don't use it
unless I absolutely need to.

Diez B. Roggisch wrote:
>
> Its a question of protocol. A iterator only guarantes that you can fetch
> items from it until it is possibly exhausted. In the same way, a stream
> may only guarantee you that you can read from it. If you need more, you
> have tgo use a different protocol. Lists are one that allows you to
> randomly access it. Files allow to seek, in addition to stream semantics.
>
> And given that python is an imperative language with side-effects, you
> don't know what sideeffects a generator can cause - which makes a
> "rewind" or copy-semantics to make it work several time impossible (in
> the general case, that is). So it has to focus on what it _can_ guarantee.
>
> Haskell and other FP languages canwork differently here, as sideeffects
> are made excplicit using e.g. monads or other means - which allows for
> re-use of a generator-like construct. But there is no way to make this
> work in pythons paradigm.
>
> Diez


 
Reply With Quote
 
bonono@gmail.com
Guest
Posts: n/a
 
      10-16-2005
thanks. I was looking for scanl in itertools but can't find it so I
implement my own then run into some subtle bugs which first made me
think my scanl is the problem. Then notice my wrong perception about
generator(and iterable in general, though the built-in iterables like
list, dict don't seem to have this characteristic).

Simon Percivall wrote:
> If you find that you want to iterate over an iterable multiple times,
> have a look at the solution that the tee() function in the itertools
> module provides (http://docs.python.org/lib/itertools-functions.html).
> (Have a look at the rest of the itertools module as well, for that
> matter.)


 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      10-16-2005
On Sun, 16 Oct 2005 15:52:54 +0200, Fredrik Lundh wrote:

> (E-Mail Removed) wrote:
>
>> I initially thought that generator/generator expression is cool (sort of
>> like the lazy evaluation in Haskell) until I notice this side effect.
>>
>> >>>a=(x for x in range(2))
>> >>>list(a)

>> [1,2]
>> >>>list(a)

>> []
>>
>> Would this make generator/generator expression's usage pretty limited ?

>
> nope.


In fairness, it is a pretty big Gotcha.



>> As when the program/system goes beyond a single module, this behaviour
>> can cause subtle bugs ?

>
> sure, in the same way as
>
> >>> f = open(filename)
> >>> f.read()

> 'hello world\n'
> >>> f.read() # oops!

> ''
>
> causes subtle bugs (that is, almost never)


Are you saying that the bugs it causes aren't subtle? *wink*


--
Steven

 
Reply With Quote
 
vivainio@gmail.com
Guest
Posts: n/a
 
      10-16-2005
> Are you saying that the bugs it causes aren't subtle? *wink*

Exactly. Destructive generator problems are caught almost immediately.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
C/C++ language proposal: Change the 'case expression' from "integral constant-expression" to "integral expression" Adem C++ 42 11-04-2008 12:39 PM
C/C++ language proposal: Change the 'case expression' from "integral constant-expression" to "integral expression" Adem C Programming 45 11-04-2008 12:39 PM
subtle class lookup with sets. cbongior@stny.rr.com Java 6 08-24-2005 09:30 PM
Subtle difference??? KeithS Firefox 2 03-02-2005 03:18 PM
The Subtle Application of Style Jo Inferis ASP .Net 1 01-11-2005 03:35 PM



Advertisments