Velocity Reviews > Re: Problem of function calls from map()

# Re: Problem of function calls from map()

Dasn
Guest
Posts: n/a

 08-22-2006

Well, please drop a glance at my current profile report:

#------------------------ test.py ---------------------
import os, sys, profile

print os.uname()
print sys.version

# size of 'dict.txt' is about 3.6M, 154563 lines
f = open('dict.txt', 'r')
print "Done."

def splitUsing(chars):
def tmp(s):
return s.split(chars)
return tmp

def sp0(lines):
"""====> sp0() -- Normal 'for' loop"""
l = []
for line in lines:
l.append(line.split('\t'))
return l

def sp1(lines):
"""====> sp1() -- List-comprehension"""
return [s.split('\t') for s in lines]

def sp2(lines):
"""====> sp2() -- Map with lambda function"""
return map(lambda s: s.split('\t'), lines)

def sp3(lines):
"""====> sp3() -- Map with splitUsing() function"""
return map(splitUsing('\t'), lines)

def sp4(lines):
"""====> sp4() -- Not correct, but very fast"""
return map(str.split, lines)

for num in xrange(5):
fname = 'sp%(num)s' % locals()
print eval(fname).__doc__
profile.run(fname+'(lines)')

#---------------------------End of test.py ----------------

\$ python test.py

('OpenBSD', 'Compaq', '3.9', 'kernel#1', 'i386')
2.4.2 (#1, Mar 2 2006, 14:17:22)
[GCC 3.3.5 (propolice)]
Done.
====> sp0() -- Normal 'for' loop
309130 function calls in 20.510 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
154563 4.160 0.000 4.160 0.000 :0(append)
1 0.010 0.010 0.010 0.010 :0(setprofile)
154563 6.490 0.000 6.490 0.000 :0(split)
1 0.380 0.380 20.500 20.500 <string>:1(?)
0 0.000 0.000 profile:0(profiler)
1 0.000 0.000 20.510 20.510 profile:0(sp0(lines))
1 9.470 9.470 20.120 20.120 test.py:20(sp0)

====> sp1() -- List-comprehension
154567 function calls in 12.240 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 :0(setprofile)
154563 6.740 0.000 6.740 0.000 :0(split)
1 0.380 0.380 12.240 12.240 <string>:1(?)
0 0.000 0.000 profile:0(profiler)
1 0.000 0.000 12.240 12.240 profile:0(sp1(lines))
1 5.120 5.120 11.860 11.860 test.py:27(sp1)

====> sp2() -- Map with lambda function
309131 function calls in 20.480 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 4.600 4.600 20.100 20.100 :0(map)
1 0.000 0.000 0.000 0.000 :0(setprofile)
154563 7.320 0.000 7.320 0.000 :0(split)
1 0.370 0.370 20.470 20.470 <string>:1(?)
0 0.000 0.000 profile:0(profiler)
1 0.010 0.010 20.480 20.480 profile:0(sp2(lines))
1 0.000 0.000 20.100 20.100 test.py:31(sp2)
154563 8.180 0.000 15.500 0.000 test.py:33(<lambda>)

====> sp3() -- Map with splitUsing() function
309132 function calls in 21.900 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 5.540 5.540 21.520 21.520 :0(map)
1 0.000 0.000 0.000 0.000 :0(setprofile)
154563 7.100 0.000 7.100 0.000 :0(split)
1 0.380 0.380 21.900 21.900 <string>:1(?)
0 0.000 0.000 profile:0(profiler)
1 0.000 0.000 21.900 21.900 profile:0(sp3(lines))
1 0.000 0.000 0.000 0.000 test.py:14(splitUsing)
154563 8.880 0.000 15.980 0.000 test.py:15(tmp)
1 0.000 0.000 21.520 21.520 test.py:35(sp3)

====> sp4() -- Not correct, but very fast
5 function calls in 3.090 CPU seconds

Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function)
1 2.660 2.660 2.660 2.660 :0(map)
1 0.000 0.000 0.000 0.000 :0(setprofile)
1 0.430 0.430 3.090 3.090 <string>:1(?)
0 0.000 0.000 profile:0(profiler)
1 0.000 0.000 3.090 3.090 profile:0(sp4(lines))
1 0.000 0.000 2.660 2.660 test.py:39(sp4)

The problem is the default behavior of str.split should be more complex
than str.split('\t'). If we could use the str.split('\t') in map(), the
result would be witty. What do u guys think?

Peter Otten
Guest
Posts: n/a

 08-22-2006
Dasn wrote:

> # size of 'dict.txt' is about 3.6M, 154563 lines
> f = open('dict.txt', 'r')

> def sp0(lines):
> ********"""====> sp0() -- Normal 'for' loop"""
> ********l = []
> ********for line in lines:
> ****************l.append(line.split('\t'))
> ********return l

Where do you get the data from in the "real world"? If it's a file you might

sp0(f)

Anyway, here's another variant you can try:

from itertools import izip, starmap, repeat

def splitlines(lines):
return list(starmap(str.split, izip(lines, repeat("\t"))))

Peter

Sion Arrowsmith
Guest
Posts: n/a

 08-22-2006
Dasn <(E-Mail Removed)> wrote:
># size of 'dict.txt' is about 3.6M, 154563 lines
>f = open('dict.txt', 'r')
>print "Done."

[ ... ]
>def sp1(lines):
> """====> sp1() -- List-comprehension"""
> return [s.split('\t') for s in lines]

[ ... ]
>def sp4(lines):
> """====> sp4() -- Not correct, but very fast"""
> return map(str.split, lines)
>
>for num in xrange(5):
> fname = 'sp%(num)s' % locals()
> print eval(fname).__doc__
> profile.run(fname+'(lines)')

>====> sp1() -- List-comprehension
> 154567 function calls in 12.240 CPU seconds

[ ... ]
>====> sp4() -- Not correct, but very fast
> 5 function calls in 3.090 CPU seconds

[ ... ]
>The problem is the default behavior of str.split should be more complex
>than str.split('\t'). If we could use the str.split('\t') in map(), the
>result would be witty. What do u guys think?

I think there's something weird going on -- sp4 should be making
154563 calls to str.split. So no wonder it goes faster -- it's
not doing any work.

How does [s.split() for s in lines] compare to sp2's
[s.split('\t') for s in lines] ?

--
\S -- http://www.velocityreviews.com/forums/(E-Mail Removed) -- http://www.chaos.org.uk/~sion/
___ | "Frankly I have no feelings towards penguins one way or the other"
\X/ | -- Arthur C. Clarke
her nu becomež se bera eadward ofdun hlęddre heafdes bęce bump bump bump

Fredrik Lundh
Guest
Posts: n/a

 08-22-2006
Sion Arrowsmith wrote:

> I think there's something weird going on -- sp4 should be making
> 154563 calls to str.split. So no wonder it goes faster -- it's not doing
> any work.

of course it does:

>>> lines = ["line\tone", "line\ttwo"]
>>> [s.split("\t") for s in lines]

[['line', 'one'], ['line', 'two']]
>>> map(str.split, lines)

[['line', 'one'], ['line', 'two']]

the problem is that he's using a Python-level profiler to benchmark things written
in C.

(you cannot really use "profile" to *benchmark* things written in Python either; the
profiler tells you where a given program spends the time, not how fast it is in com-
parision with other programs)

</F>

Sion Arrowsmith
Guest
Posts: n/a

 08-22-2006
Fredrik Lundh <(E-Mail Removed)> wrote:
>Sion Arrowsmith wrote:
>> I think there's something weird going on -- sp4 should be making
>> 154563 calls to str.split. So no wonder it goes faster -- it's not doing
>> any work.

>the problem is that he's using a Python-level profiler to benchmark things written
>in C.
>
>(you cannot really use "profile" to *benchmark* things written in Python either; the
>profiler tells you where a given program spends the time, not how fast it is in com-
>parision with other programs)

Hmm. Playing around with timeit suggests that although split() *is*
faster than split("\t"), it's fractional, rather than the OP's four
times faster. Is the overhead of profile keeping track of calls in
Python getting in the way? (Not having used profile -- hence my
confusion.) And why can map() keep everything at the C level when
the list comprehension can't?

--
\S -- (E-Mail Removed) -- http://www.chaos.org.uk/~sion/
___ | "Frankly I have no feelings towards penguins one way or the other"
\X/ | -- Arthur C. Clarke
her nu becomež se bera eadward ofdun hlęddre heafdes bęce bump bump bump

Fredrik Lundh
Guest
Posts: n/a

 08-22-2006
Sion Arrowsmith wrote:

>> (you cannot really use "profile" to *benchmark* things written in Python either; the
>> profiler tells you where a given program spends the time, not how fast it is in com-
>> parision with other programs)

>
> Hmm. Playing around with timeit suggests that although split() *is*
> faster than split("\t"), it's fractional, rather than the OP's four
> times faster. Is the overhead of profile keeping track of calls in
> Python getting in the way?

correct.

> And why can map() keep everything at the C level when the list com-
> prehension can't?

map is called with two Python objects (the str.split callable and the
sequence object), while the list comprehension is turned into a byte-
code loop that evaluates s.split for each item in the sequence; compare
and contrast:

>>> def func(a):

.... return map(str.split, a)
....
>>> dis.dis(func)

12 CALL_FUNCTION 2
15 RETURN_VALUE

>>> def func(a):

.... return [s.split() for s in a]
....
>>> dis.dis(func)

2 0 BUILD_LIST 0
3 DUP_TOP
4 STORE_FAST 1 (_[1])
10 GET_ITER
>> 11 FOR_ITER 19 (to 33)

14 STORE_FAST 2 (s)
26 CALL_FUNCTION 0
29 LIST_APPEND
30 JUMP_ABSOLUTE 11
>> 33 DELETE_FAST 1 (_[1])

36 RETURN_VALUE

a local variable using an integer index)

</F>

Steve Holden
Guest
Posts: n/a

 08-23-2006
Fredrik Lundh wrote:
> Sion Arrowsmith wrote:
>
>
>>>(you cannot really use "profile" to *benchmark* things written in Python either; the
>>>profiler tells you where a given program spends the time, not how fast it is in com-
>>>parision with other programs)

>>
>>Hmm. Playing around with timeit suggests that although split() *is*
>>faster than split("\t"), it's fractional, rather than the OP's four
>>times faster. Is the overhead of profile keeping track of calls in
>>Python getting in the way?

>
>
> correct.
>
>
>>And why can map() keep everything at the C level when the list com-

>
> > prehension can't?

>
> map is called with two Python objects (the str.split callable and the
> sequence object), while the list comprehension is turned into a byte-
> code loop that evaluates s.split for each item in the sequence; compare
> and contrast:
>
> >>> def func(a):

> .... return map(str.split, a)
> ....
> >>> dis.dis(func)

> 2 0 LOAD_GLOBAL 0 (map)
> 12 CALL_FUNCTION 2
> 15 RETURN_VALUE
>
> >>> def func(a):

> .... return [s.split() for s in a]
> ....
> >>> dis.dis(func)

> 2 0 BUILD_LIST 0
> 3 DUP_TOP
> 4 STORE_FAST 1 (_[1])
> 10 GET_ITER
> >> 11 FOR_ITER 19 (to 33)

> 14 STORE_FAST 2 (s)
> 26 CALL_FUNCTION 0
> 29 LIST_APPEND
> 30 JUMP_ABSOLUTE 11
> >> 33 DELETE_FAST 1 (_[1])

> 36 RETURN_VALUE
>
> a local variable using an integer index)
>
> </F>
>

Well I guess if people wanted to argue for keeping the functionals this
should be on the list ...

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

Fredrik Lundh
Guest
Posts: n/a

 08-24-2006
Steve Holden wrote:

> Well I guess if people wanted to argue for keeping the functionals this
> should be on the list ...

who's arguing ?

is this perhaps a little like the "now that we have lexical scoping, the default
argument object binding trick is no longer needed" myth ?

</F>

Steve Holden
Guest
Posts: n/a

 08-28-2006
Fredrik Lundh wrote:
> Steve Holden wrote:
>
>
>>Well I guess if people wanted to argue for keeping the functionals this
>>should be on the list ...

>
>
> who's arguing ?
>

Please note that word "if", but you are surely aware that there havebeen
suggestions that Python's functional programming aspects aren't
mainstream and should therefore be ditched.

> is this perhaps a little like the "now that we have lexical scoping, the default
> argument object binding trick is no longer needed" myth ?
>

Could be. Removing that feature would also be a mistake.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

Dasn
Guest
Posts: n/a

 08-28-2006
On Tue, Aug 22, 2006 at 04:50:39PM +0200, Fredrik Lundh wrote:
> (you cannot really use "profile" to *benchmark* things written in
> Python either; the profiler tells you where a given program spends the
> time, not how fast it is in com- parision with other programs)