«

Hi,

When list comprehension was added to the language, I had a lot of
trouble understanding it but now that I am familiar with it, I am not
sure how I programmed in Python without it.

Now I see that generator expressions have been added to the language
with 2.4 and I question the need for it. I know that it allows for lazy
evaluation which speeds things up for larger lists but why was it
necessary to add it instead of improving list comprehension?

Was there some sort of limitation that prevented list comprehension from
taking over the functionality that generator expressions provide?

I have always liked the fact that Python has limited capabilities for
having more than one way to do it and it seem to me that generator
expressions break that philosophy. It is similar to how you have to use
range or xrange depending on how large the range is.

Can someone please explain the reasoning behind it?

Thanks,
Mahesh

Mahesh Padmanabhan <> wrote:

> Hi,
>
> When list comprehension was added to the language, I had a lot of
> trouble understanding it but now that I am familiar with it, I am not
> sure how I programmed in Python without it.

Oh good.

>
> Now I see that generator expressions have been added to the language
> with 2.4 and I question the need for it. I know that it allows for lazy
> evaluation which speeds things up for larger lists but why was it
> necessary to add it instead of improving list comprehension?
>
> Was there some sort of limitation that prevented list comprehension from
> taking over the functionality that generator expressions provide?

Sure, and that limitation is: list comprehensions return lists. This
one "limitation" (together with Python's applicative order evaluation,
and you couldn't change THAT without breaking the whole caboodle of
existing programs!) implies everything else.

>
> I have always liked the fact that Python has limited capabilities for
> having more than one way to do it and it seem to me that generator
> expressions break that philosophy. It is similar to how you have to use
> range or xrange depending on how large the range is.
>
> Can someone please explain the reasoning behind it?

Generator comprehensions are wonderful and there is no way Python list
comprehensions can provide the same features, since lists need to be
lists. Sure, list(e(x) for x in foo) IS just the same thing as [e(x)
for x in foo]. We'll remove the redundancy in 3.0 -- not earlier
because it will break backwards compatibility. The only sensible way I
can see right now for 3.0 to remove this redundancy is by removing list
comprehensions and leaving only generator comprehensions, btw.


Alex

In article <1gj8y77.6qshr41q531veN%>,
(Alex Martelli) wrote:


> Sure, and that limitation is: list comprehensions return lists. This
> one "limitation" (together with Python's applicative order evaluation,
> and you couldn't change THAT without breaking the whole caboodle of
> existing programs!) implies everything else.

Is returning a list really a limitation considering that lists can be
transformed quite easily?

What is "Python's applicative order evaluation" and how do generator
expressions get around it?


> Generator comprehensions are wonderful and there is no way Python list
> comprehensions can provide the same features, since lists need to be
> lists. Sure, list(e(x) for x in foo) IS just the same thing as [e(x)
> for x in foo]. We'll remove the redundancy in 3.0 -- not earlier
> because it will break backwards compatibility. The only sensible way I
> can see right now for 3.0 to remove this redundancy is by removing list
> comprehensions and leaving only generator comprehensions, btw.

I am still not clear of the advantages of using generator expressions
(other than less memory consumption) instead of list comprehension for
any given class of problems. Can you cite concrete use cases where
generator expressions would be preferred over list comprehension?

Thanks,
Mahesh

Mahesh Padmanabhan <> writes:
> I am still not clear of the advantages of using generator expressions
> (other than less memory consumption)

Why do you think using bounded rather than unbounded amounts of memory
is an insignificant advantage?

Mahesh Padmanabhan <> wrote:
>
> I am still not clear of the advantages of using generator expressions
> (other than less memory consumption) instead of list comprehension for
> any given class of problems. Can you cite concrete use cases where
> generator expressions would be preferred over list comprehension?

Here's a simple example

linecount = sum(1 for line in file)
linecount = sum([1 for line in file])

The former requires an in-memory list equal to the size of the file.
Consigning "other than memory consumption" to a parenthetical remark
misses its importance.

martin

Mahesh Padmanabhan wrote:

> In article <1gj8y77.6qshr41q531veN%>,
> (Alex Martelli) wrote:
>
>
>> Sure, and that limitation is: list comprehensions return lists. This
>> one "limitation" (together with Python's applicative order evaluation,
>> and you couldn't change THAT without breaking the whole caboodle of
>> existing programs!) implies everything else.
>
> Is returning a list really a limitation considering that lists can be
> transformed quite easily?
>
> What is "Python's applicative order evaluation" and how do generator
> expressions get around it?
>
>
>> Generator comprehensions are wonderful and there is no way Python list
>> comprehensions can provide the same features, since lists need to be
>> lists. Sure, list(e(x) for x in foo) IS just the same thing as [e(x)
>> for x in foo]. We'll remove the redundancy in 3.0 -- not earlier
>> because it will break backwards compatibility. The only sensible way I
>> can see right now for 3.0 to remove this redundancy is by removing list
>> comprehensions and leaving only generator comprehensions, btw.
>
> I am still not clear of the advantages of using generator expressions
> (other than less memory consumption) instead of list comprehension for
> any given class of problems. Can you cite concrete use cases where
> generator expressions would be preferred over list comprehension?
>
> Thanks,
> Mahesh

- lower memory usage
- lower CPU usage ( sometimes, you don't need to expand the whole generator,
see below )
- ability to manipulate infinite generators

Is that enouth ?

On Sat, 28 Aug 2004 14:09:34 -0600, Mahesh Padmanabhan wrote:
> Is returning a list really a limitation considering that lists can be
> transformed quite easily?

http://aspn.activestate.com/ASPN/Coo.../Recipe/190465

It is trivial to generate huge lists and more people need them than you
might think. Not choking the VM system with multi-hundred megabytes in
allocations is a non-trivial performance advantage, especially as the
alternative may be not finishing at all.

You, by the way, have it backwards. It is trivial to convert a generator
into a list, without advantage loss. The opposite is not possible. That's
why Alex, elsewhere in the newsgroup, is saying list comprehensions might
someday be replaced by list(x for x in whatever), but you're never going
to see him say generator expressions are adequately captured by iter([x
for x in whatever]).

In article <mahesh->,
Mahesh Padmanabhan <> wrote:
>I am still not clear of the advantages of using generator expressions
>(other than less memory consumption) instead of list comprehension for
>any given class of problems. Can you cite concrete use cases where
>generator expressions would be preferred over list comprehension?

Storage economy and the time saved by omitting lots of
object creation and destruction are the main advantages of
generator expressions.

The advantage of lists is that they'll stand still while
you do things to them (things like iterating over them in
many and novel ways.) The mind does boggle at the idea of a
data-processing program that has replaced all containers
with generator expressions, and creates all its information
on-demand without storing anything anywhere.

Regards. Mel.

>>>>> "Mahesh" == Mahesh Padmanabhan <> writes:

Mahesh> Is returning a list really a limitation considering that
Mahesh> lists can be transformed quite easily?

Yes. (1) You lose your memory, and (2) you can't use whatever you get
outside after the list comprehension hidden loop when evaluating it.
Let's have some examples.

You lose you memory because you have to generate a whole list, which
might be unnecessary because your processing eventually don't need it.
It is the same as the difference between range() and xrange(). E.g.,
you might write

for a in [x*x for x in range(100000)]:
print a

you have to wait until you generate the 100k list, and at that time
you start printing out x*x for each of the values. The middle-ground

for a in [x*x for x in xrange(100000)]:
print a

save half the memory, but still needs to generate the 100k list, and
you have to wait a long time before you print the first result. Once
you get generator expression, you can say

for a in (x*x for x in xrange(100000)):
print a

and it will do the same thing, except that you don't need to wait at
the beginning, and at no instant there is a 100000 element list
sitting in memory.

[N.B.: Of course, in this simple example you'll instead write

for x in xrange(100000):
print x*x

but sometimes things are not as easy, e.g., what you'd do if you have
to pass the abstraction "x*x in xrange(100000)" into a function as a
function argument?]

Now let's turn to the second point: with list comprehension, the
evaluation of an element happens after the evaluation of previous
elements, but before the previous elements are being used. At times
this makes things hard to achieve or even impossible (and at that time
you have to throw up your hands and write a generator function
instead). E.g., suppose you have this:

import time
hash = {}
def process1(x):
for i in xrange(1, 11):
hash[x * i] = 1
def process2(x):
for i in xrange(2, 12):
hash[x * i] = 1
if time.time() % 2:
process = process1
else:
process = process2

Now you have a loop, which you want a neater way to rewrite:

for x in xrange(1000):
if not hash.has_key(x):
process(x * x)

in such a way that you don't need to let others specify the exact loop
to run. Intuitively you'd like to write a list comprehension to do
that. So you'd like to write

for y in [x*x for x in xrange(1000) if not hash.has_key(x)]:
process(y)

and let others pass the list into the function. But this makes
hash.hash_key(x) to be called when none of the process(y) is called,
so it breaks completely. With generator expression, you write:

for y in (x*x for x in xrange(1000) if not hash.has_key(x)):
process(y)

which do the trick. Note that now "x*x for x in xrange(1000) if not
hash.has_key(x)" is an object, and you can move it out of the
function, ask somebody else to pass it to you---which you can't do in
the original for loop. I.e., now you can say

def func(gen):
for x in gen:
process(x)

and let somebody call

func(x*x for x in xrange(1000 if not hash.has_key(x)))

Without generator expression, to achieve this you must code a
generator function. So generator expression helps you to write simple
generators.

Regards,
Isaac.

In article <>,
Isaac To <> wrote:

[snip]

Thanks Isaac for providing such a detailed explanation. I understand now
why generator expressions make more sense.

I just wish that generator expressions had been designed into the
language without going the way of list comprehension -> generator
expressions.