Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Question regarding Higher-Order-Programming in Python

Reply
Thread Tools

Question regarding Higher-Order-Programming in Python

 
 
Mark Fink
Guest
Posts: n/a
 
      12-22-2010
I am about to learn Higher-Order-Programming with Lisp, Haskell, and
Python. Some people who have gone completely out of their mind call
this FP.

In Haskell I learned that when I use map on a list it starts nesting
as soon as I start adding elements. If I do not like the nesting I use
ConcatMap.

In Python I have a similar scenario. I have a generator which creates
some combinatorics of a input dictionary. The list starts nesting. Now
I could use reduce(concat, ...) which would be the correct thing from
a functional perspective. But from a resource utilization perspective
it is not a good idea since the reduce is executing the generator.

I tried to illustrate this using a small example (the often in
combinatorics the real thing would be much bigger that is why I need
to use a generator):

>>> from operator import itemgetter, concat
>>> import itertools as it
>>> from functools import partial
>>>
>>> dims = {'number': [1,2,3], 'letter': ['a', 'b'], 'special': ['+', '-']}
>>> dims

{'special': ['+', '-'], 'number': [1, 2, 3], 'letter': ['a', 'b']}
>>> def get_products(keys):

.... # helper to get products from keys in the following form:
.... # [('bold', True), ('color', 'black')]
.... values = itemgetter(*keys)(dims)
.... product = it.product(*values)
.... return map(partial(zip, keys), product)
....
>>> comb = it.combinations(dims, 2)
>>> comb_l = list(comb)
>>> comb_l

[('special', 'number'), ('special', 'letter'), ('number', 'letter')]
>>> res = map(get_products, comb_l)
>>> res

[[[('special', '+'), ('number', 1)], [('special', '+'), ('number',
2)], [('special', '+'), ('number', 3)], [('special', '-'), ('number',
1)], [('special', '-'), ('number', 2)], [('special', '-'), ('number',
3)]], [[('special', '+'), ('letter', 'a')], [('special', '+'),
('letter', 'b')], [('special', '-'), ('letter', 'a')], [('special',
'-'), ('letter', 'b')]], [[('number', 1), ('letter', 'a')],
[('number', 1), ('letter', 'b')], [('number', 2), ('letter', 'a')],
[('number', 2), ('letter', 'b')], [('number', 3), ('letter', 'a')],
[('number', 3), ('letter', 'b')]]] # the resulting list is nested one
level to deep caused by the map(get_products, ..
>>>


My problem is that I want to get single elements from the generator
like [('special', '+'), ('number', 1)]. But this does not work because
the list is now nested to deep.

That is what I expect: (I could get something like that with the
following >>> res = reduce(concat, res)
[[('special', '+'), ('number', 1)], [('special', '+'), ('number', 2)],
[('special', '+'), ('number', 3)], [('special', '-'), ('number', 1)],
[('special', '-'), ('number', 2)], [('special', '-'), ('number', 3)],
[('special', '+'), ('letter', 'a')], [('special', '+'), ('letter',
'b')], [('special', '-'), ('letter', 'a')], [('special', '-'),
('letter', 'b')], [('number', 1), ('letter', 'a')], [('number', 1),
('letter', 'b')], [('number', 2), ('letter', 'a')], [('number', 2),
('letter', 'b')], [('number', 3), ('letter', 'a')], [('number', 3),
('letter', 'b')]]

I have seen the problem many times but so far I could not google a
solution on the web.

By the way do you know any substantial example using FP in Python
(generators, imap, ifilter, ...)?
 
Reply With Quote
 
 
 
 
Peter Otten
Guest
Posts: n/a
 
      12-22-2010
Mark Fink wrote:

> I am about to learn Higher-Order-Programming with Lisp, Haskell, and
> Python. Some people who have gone completely out of their mind call
> this FP.
>
> In Haskell I learned that when I use map on a list it starts nesting
> as soon as I start adding elements. If I do not like the nesting I use
> ConcatMap.
>
> In Python I have a similar scenario. I have a generator which creates
> some combinatorics of a input dictionary. The list starts nesting. Now
> I could use reduce(concat, ...) which would be the correct thing from
> a functional perspective. But from a resource utilization perspective
> it is not a good idea since the reduce is executing the generator.
>
> I tried to illustrate this using a small example (the often in
> combinatorics the real thing would be much bigger that is why I need
> to use a generator):
>
>>>> from operator import itemgetter, concat
>>>> import itertools as it
>>>> from functools import partial
>>>>
>>>> dims = {'number': [1,2,3], 'letter': ['a', 'b'], 'special': ['+', '-']}
>>>> dims

> {'special': ['+', '-'], 'number': [1, 2, 3], 'letter': ['a', 'b']}
>>>> def get_products(keys):

> ... # helper to get products from keys in the following form:
> ... # [('bold', True), ('color', 'black')]
> ... values = itemgetter(*keys)(dims)
> ... product = it.product(*values)
> ... return map(partial(zip, keys), product)
> ...
>>>> comb = it.combinations(dims, 2)
>>>> comb_l = list(comb)
>>>> comb_l

> [('special', 'number'), ('special', 'letter'), ('number', 'letter')]
>>>> res = map(get_products, comb_l)
>>>> res

> [[[('special', '+'), ('number', 1)], [('special', '+'), ('number',
> 2)], [('special', '+'), ('number', 3)], [('special', '-'), ('number',
> 1)], [('special', '-'), ('number', 2)], [('special', '-'), ('number',
> 3)]], [[('special', '+'), ('letter', 'a')], [('special', '+'),
> ('letter', 'b')], [('special', '-'), ('letter', 'a')], [('special',
> '-'), ('letter', 'b')]], [[('number', 1), ('letter', 'a')],
> [('number', 1), ('letter', 'b')], [('number', 2), ('letter', 'a')],
> [('number', 2), ('letter', 'b')], [('number', 3), ('letter', 'a')],
> [('number', 3), ('letter', 'b')]]] # the resulting list is nested one
> level to deep caused by the map(get_products, ..
>>>>

>
> My problem is that I want to get single elements from the generator
> like [('special', '+'), ('number', 1)]. But this does not work because
> the list is now nested to deep.


Like this?

>>> [(k, v) for k, vv in dims.iteritems() for v in vv]

[('special', '+'), ('special', '-'), ('number', 1), ('number', 2),
('number', 3), ('letter', 'a'), ('letter', 'b')]

But these are just glorified for-loops, so:

>>> list(chain.from_iterable(starmap(product, izip(izip(dims.iterkeys()),

dims.itervalues()))))
[('special', '+'), ('special', '-'), ('number', 1), ('number', 2),
('number', 3), ('letter', 'a'), ('letter', 'b')]

Peter

 
Reply With Quote
 
 
 
 
Mark Fink
Guest
Posts: n/a
 
      12-22-2010
> >>> list(chain.from_iterable(starmap(product, izip(izip(dims.iterkeys()),
>
> dims.itervalues()))))
> [('special', '+'), ('special', '-'), ('number', 1), ('number', 2),
> ('number', 3), ('letter', 'a'), ('letter', 'b')]
>
> Peter


so far I have never noticed chain.from_iterable, but many thanks to
you Peter, I have now a beautiful solution to this problem.
>>> from itertools import chain
>>> comb = it.combinations(dims, 2)
>>> l = chain.from_iterable(it.imap(get_products, comb))
>>> l.next()

[('special', '+'), ('number', 1)]
>>> l.next()

[('special', '+'), ('number', 2)]
 
Reply With Quote
 
Arnaud Delobelle
Guest
Posts: n/a
 
      12-22-2010
Mark Fink <(E-Mail Removed)> writes:

> so far I have never noticed chain.from_iterable, but many thanks to
> you Peter, I have now a beautiful solution to this problem.
>>>> from itertools import chain
>>>> comb = it.combinations(dims, 2)
>>>> l = chain.from_iterable(it.imap(get_products, comb))


You can also write this as:

l = (p for c in comb for p in get_products(c))

>>>> l.next()

> [('special', '+'), ('number', 1)]
>>>> l.next()

> [('special', '+'), ('number', 2)]


Also in your original post you define get_products:

>>>> def get_products(keys):

> ... # helper to get products from keys in the following form:
> ... # [('bold', True), ('color', 'black')]
> ... values = itemgetter(*keys)(dims)
> ... product = it.product(*values)
> ... return map(partial(zip, keys), product)
> ...


You could define it as e.g.

def get_products(keys):
key_values = [[(k, v) for v in values] for k in keys]
return it.product(*key_values)

But maybe for some reason you are trying to avoid list comprehensions
and generator expressions?

--
Arnaud
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Doubt regarding python Compilation praba kar Python 1 05-01-2005 07:00 AM
query regarding python script Shubhra Gupta Python 3 05-20-2004 02:32 PM
Help regarding damaged Monty Python DVD? Bex DVD Video 5 04-16-2004 01:11 AM
query regarding embeding python in C Vardhman Jain Python 1 04-08-2004 03:33 AM
funny slashdot quote regarding python 2.3 release Guyon Morée Python 4 07-30-2003 10:24 PM



Advertisments