Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Filter versus comprehension (was Re: something about split()???)

Reply
Thread Tools

Filter versus comprehension (was Re: something about split()???)

 
 
Terry Reedy
Guest
Posts: n/a
 
      08-22-2012
On 8/22/2012 3:30 AM, Mark Lawrence wrote:
> On 22/08/2012 06:46, Terry Reedy wrote:
>> On 8/21/2012 11:43 PM, mingqiang hu wrote:
>>> why filter is bad when use lambda ?

>>
>> Inefficient, not 'bad'. Because the equivalent comprehension or
>> generator expression does not require a function call.


for each item in the iterable.

> A case of premature optimisation?


No, as regards my post. I simply made a factual statement without
advocating a particular action.

filter(lambda x: <expr>, iterable)
(x for x in iterable if <expr>)

both create iterators that produce the items in iterable such that
bool(<expr>) is true. The following, with output rounded, shows
something of the effect of the extra function call.

>>> timeit.timeit("list(i for i in ranger if False)", "ranger=range(0)")

0.91
>>> timeit.timeit("list(i for i in ranger if False)", "ranger=range(20)")

1.28
>>> timeit.timeit("list(filter(lambda i: False, ranger))",

"ranger=range(0)")
0.83
>>> timeit.timeit("list(filter(lambda i: False, ranger))",

"ranger=range(20)")
2.60

Simply keeping true items is faster with filter -- at least on my
particular machine with 3.3.0b2.

>>> timeit.timeit("list(filter(None, ranger))", "ranger=range(20)")

1.03

Filter is also faster if the expression is a function call.

>>> timeit.timeit("list(filter(f, ranger))", "ranger=range(20);

f=lambda i: False")
2.5033614114454394
>>> timeit.timeit("list(i for i in ranger if f(i))", "ranger=range(20);

f=lambda i: False")
3.2394095327040304

---
Perhaps or even yes as regards the so-called rule 'always use
comprehension'. If one prefers filter as more readable, if one only
wants to keep true items, if the expression is a function call, if
evaluating the expression takes much more time than the extra function
call so the latter does not matter, if the number of items is few enough
that the extra time does not matter, then the rule is not needed or even
wrong.

So I think PyLint should be changed to stop its filter fud.

--
Terry Jan Reedy

 
Reply With Quote
 
 
 
 
Ramchandra Apte
Guest
Posts: n/a
 
      08-24-2012
On Wednesday, 22 August 2012 22:13:04 UTC+5:30, Terry Reedy wrote:
> On 8/22/2012 3:30 AM, Mark Lawrence wrote:
>
> > On 22/08/2012 06:46, Terry Reedy wrote:

>
> >> On 8/21/2012 11:43 PM, mingqiang hu wrote:

>
> >>> why filter is bad when use lambda ?

>
> >>

>
> >> Inefficient, not 'bad'. Because the equivalent comprehension or

>
> >> generator expression does not require a function call.

>
>
>
> for each item in the iterable.
>
>
>
> > A case of premature optimisation?

>
>
>
> No, as regards my post. I simply made a factual statement without
>
> advocating a particular action.
>
>
>
> filter(lambda x: <expr>, iterable)
>
> (x for x in iterable if <expr>)
>
>
>
> both create iterators that produce the items in iterable such that
>
> bool(<expr>) is true. The following, with output rounded, shows
>
> something of the effect of the extra function call.
>
>
>
> >>> timeit.timeit("list(i for i in ranger if False)", "ranger=range(0)")

>
> 0.91
>
> >>> timeit.timeit("list(i for i in ranger if False)", "ranger=range(20)")

>
> 1.28
>
> >>> timeit.timeit("list(filter(lambda i: False, ranger))",

>
> "ranger=range(0)")
>
> 0.83
>
> >>> timeit.timeit("list(filter(lambda i: False, ranger))",

>
> "ranger=range(20)")
>
> 2.60
>
>
>
> Simply keeping true items is faster with filter -- at least on my
>
> particular machine with 3.3.0b2.
>
>
>
> >>> timeit.timeit("list(filter(None, ranger))", "ranger=range(20)")

>
> 1.03
>
>
>
> Filter is also faster if the expression is a function call.
>
>
>
> >>> timeit.timeit("list(filter(f, ranger))", "ranger=range(20);

>
> f=lambda i: False")
>
> 2.5033614114454394
>
> >>> timeit.timeit("list(i for i in ranger if f(i))", "ranger=range(20);

>
> f=lambda i: False")
>
> 3.2394095327040304
>
>
>
> ---
>
> Perhaps or even yes as regards the so-called rule 'always use
>
> comprehension'. If one prefers filter as more readable, if one only
>
> wants to keep true items, if the expression is a function call, if
>
> evaluating the expression takes much more time than the extra function
>
> call so the latter does not matter, if the number of items is few enough
>
> that the extra time does not matter, then the rule is not needed or even
>
> wrong.
>
>
>
> So I think PyLint should be changed to stop its filter fud.
>
>
>
> --
>
> Terry Jan Reedy


When filtering for true values, filter(None,xxx) can be used
Your examples with lambda i:False are unrealistic - you are comparing `if False` vs <lambda function>(xx) - function call vs boolean check
 
Reply With Quote
 
 
 
 
Ramchandra Apte
Guest
Posts: n/a
 
      08-24-2012
On Wednesday, 22 August 2012 22:13:04 UTC+5:30, Terry Reedy wrote:
> On 8/22/2012 3:30 AM, Mark Lawrence wrote:
>
> > On 22/08/2012 06:46, Terry Reedy wrote:

>
> >> On 8/21/2012 11:43 PM, mingqiang hu wrote:

>
> >>> why filter is bad when use lambda ?

>
> >>

>
> >> Inefficient, not 'bad'. Because the equivalent comprehension or

>
> >> generator expression does not require a function call.

>
>
>
> for each item in the iterable.
>
>
>
> > A case of premature optimisation?

>
>
>
> No, as regards my post. I simply made a factual statement without
>
> advocating a particular action.
>
>
>
> filter(lambda x: <expr>, iterable)
>
> (x for x in iterable if <expr>)
>
>
>
> both create iterators that produce the items in iterable such that
>
> bool(<expr>) is true. The following, with output rounded, shows
>
> something of the effect of the extra function call.
>
>
>
> >>> timeit.timeit("list(i for i in ranger if False)", "ranger=range(0)")

>
> 0.91
>
> >>> timeit.timeit("list(i for i in ranger if False)", "ranger=range(20)")

>
> 1.28
>
> >>> timeit.timeit("list(filter(lambda i: False, ranger))",

>
> "ranger=range(0)")
>
> 0.83
>
> >>> timeit.timeit("list(filter(lambda i: False, ranger))",

>
> "ranger=range(20)")
>
> 2.60
>
>
>
> Simply keeping true items is faster with filter -- at least on my
>
> particular machine with 3.3.0b2.
>
>
>
> >>> timeit.timeit("list(filter(None, ranger))", "ranger=range(20)")

>
> 1.03
>
>
>
> Filter is also faster if the expression is a function call.
>
>
>
> >>> timeit.timeit("list(filter(f, ranger))", "ranger=range(20);

>
> f=lambda i: False")
>
> 2.5033614114454394
>
> >>> timeit.timeit("list(i for i in ranger if f(i))", "ranger=range(20);

>
> f=lambda i: False")
>
> 3.2394095327040304
>
>
>
> ---
>
> Perhaps or even yes as regards the so-called rule 'always use
>
> comprehension'. If one prefers filter as more readable, if one only
>
> wants to keep true items, if the expression is a function call, if
>
> evaluating the expression takes much more time than the extra function
>
> call so the latter does not matter, if the number of items is few enough
>
> that the extra time does not matter, then the rule is not needed or even
>
> wrong.
>
>
>
> So I think PyLint should be changed to stop its filter fud.
>
>
>
> --
>
> Terry Jan Reedy


When filtering for true values, filter(None,xxx) can be used
Your examples with lambda i:False are unrealistic - you are comparing `if False` vs <lambda function>(xx) - function call vs boolean check
 
Reply With Quote
 
Terry Reedy
Guest
Posts: n/a
 
      08-24-2012
On 8/24/2012 10:44 AM, Ramchandra Apte wrote:
> On Wednesday, 22 August 2012 22:13:04 UTC+5:30, Terry Reedy wrote:


>> >>> timeit.timeit("list(i for i in ranger if False)", "ranger=range(0)")

>>
>> 0.91
>>
>> >>> timeit.timeit("list(i for i in ranger if False)", "ranger=range(20)")

>>
>> 1.28
>>
>> >>> timeit.timeit("list(filter(lambda i: False, ranger))",

>>
>> "ranger=range(0)")
>>
>> 0.83
>>
>> >>> timeit.timeit("list(filter(lambda i: False, ranger))",

>>
>> "ranger=range(20)")
>>
>> 2.60


Your mail agent in inserting blank lines in quotes -- google?
See if you can turn that off.

> Your examples with lambda i:False are unrealistic - you are comparing
> `if False` vs <lambda function>(xx) - function call vs boolean check


That is exactly the comparison I wanted to make. The iteration + boolean
check takes .37 for 20 items, the iteration + call 1.77.

--
Terry Jan Reedy

 
Reply With Quote
 
Dennis Lee Bieber
Guest
Posts: n/a
 
      08-24-2012
On Fri, 24 Aug 2012 12:04:54 -0400, Terry Reedy <(E-Mail Removed)>
declaimed the following in gmane.comp.python.general:


>
> Your mail agent in inserting blank lines in quotes -- google?
> See if you can turn that off.
>

It appears to be a change Google made in the last month or two... My
hypothesis is that they are replacing hard EOL found in inbound NNTP
with an HTML <p>, and then on outgoing replacing the <p> with a pair of
NNTP line endings. In contrast, text composed on Google is coming in as
long single lines (since quoting said text in a response produces on a
">" at the start of the paragraph.
--
Wulfraed Dennis Lee Bieber AF6VN
http://www.velocityreviews.com/forums/(E-Mail Removed) HTTP://wlfraed.home.netcom.com/

 
Reply With Quote
 
Walter Hurry
Guest
Posts: n/a
 
      08-24-2012
On Fri, 24 Aug 2012 14:29:00 -0400, Dennis Lee Bieber wrote:

> It appears to be a change Google made in the last month or two... My
> hypothesis is that they are replacing hard EOL found in inbound NNTP
> with an HTML <p>, and then on outgoing replacing the <p> with a pair of
> NNTP line endings. In contrast, text composed on Google is coming in as
> long single lines (since quoting said text in a response produces on a
> ">" at the start of the paragraph.


Google Groups sucks. These are computer literate people here. Why don't
they just use a proper newsreader?
 
Reply With Quote
 
Dennis Lee Bieber
Guest
Posts: n/a
 
      08-24-2012
On Fri, 24 Aug 2012 19:03:51 +0000 (UTC), Walter Hurry
<(E-Mail Removed)> declaimed the following in
gmane.comp.python.general:

>
> Google Groups sucks. These are computer literate people here. Why don't
> they just use a proper newsreader?


Probably because their ISP doesn't offer a free server <G>

--
Wulfraed Dennis Lee Bieber AF6VN
(E-Mail Removed) HTTP://wlfraed.home.netcom.com/

 
Reply With Quote
 
Terry Reedy
Guest
Posts: n/a
 
      08-24-2012
On 8/24/2012 5:56 PM, Dennis Lee Bieber wrote:
> On Fri, 24 Aug 2012 19:03:51 +0000 (UTC), Walter Hurry
> <(E-Mail Removed)> declaimed the following in
> gmane.comp.python.general:
>
>>
>> Google Groups sucks. These are computer literate people here. Why don't
>> they just use a proper newsreader?

>
> Probably because their ISP doesn't offer a free server <G>


Python lists are available on the free gmane mail-to-news server.



--
Terry Jan Reedy

 
Reply With Quote
 
Emile van Sebille
Guest
Posts: n/a
 
      08-24-2012
On 8/24/2012 3:03 PM Terry Reedy said...
> On 8/24/2012 5:56 PM, Dennis Lee Bieber wrote:
>> On Fri, 24 Aug 2012 19:03:51 +0000 (UTC), Walter Hurry
>> <(E-Mail Removed)> declaimed the following in
>> gmane.comp.python.general:
>>
>>>
>>> Google Groups sucks. These are computer literate people here. Why don't
>>> they just use a proper newsreader?

>>
>> Probably because their ISP doesn't offer a free server <G>

>
> Python lists are available on the free gmane mail-to-news server.


I'm getting high load related denials with the gmane connections a lot
recently so I'm open to alternatives.

Suggestions or recommendations?


Emile



 
Reply With Quote
 
Mark Lawrence
Guest
Posts: n/a
 
      08-24-2012
On 24/08/2012 23:03, Terry Reedy wrote:
> On 8/24/2012 5:56 PM, Dennis Lee Bieber wrote:
>> On Fri, 24 Aug 2012 19:03:51 +0000 (UTC), Walter Hurry
>> <(E-Mail Removed)> declaimed the following in
>> gmane.comp.python.general:
>>
>>>
>>> Google Groups sucks. These are computer literate people here. Why don't
>>> they just use a proper newsreader?

>>
>> Probably because their ISP doesn't offer a free server <G>

>
> Python lists are available on the free gmane mail-to-news server.
>


I don't think the core-mentorship list is available on gmane. Have I
missed it, has nobody asked for it to go on there or what?


--
Cheers.

Mark Lawrence.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: looping versus comprehension Chris Angelico Python 2 01-31-2013 08:28 AM
looping versus comprehension Robin Becker Python 0 01-30-2013 02:58 PM
List comprehension in if clause of another list comprehension Vedran Furac( Python 4 12-19-2008 01:35 PM
Re: Mozilla versus IE versus Opera versus Safari Peter Potamus the Purple Hippo Firefox 0 05-08-2008 12:56 PM
equal? versus eql? versus == versus === verus <=> Paul Butcher Ruby 12 11-28-2007 06:06 AM



Advertisments