Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > How do I iterate over items in a dict grouped by N number ofelements?

Reply
Thread Tools

How do I iterate over items in a dict grouped by N number ofelements?

 
 
Noah
Guest
Posts: n/a
 
      03-14-2008
What is the fastest way to select N items at a time from a dictionary?
I'm iterating over a dictionary of many thousands of items.
I want to operate on only 100 items at a time.
I want to avoid copying items using any sort of slicing.
Does itertools copy items?

This works, but is ugly:

>>> from itertools import *
>>> D = {'a':1, 'b':2, 'c':3, 'd':4, 'e':5, 'f':6, 'g':7, 'h':8, 'i':9, 'j':10}
>>> N = 3
>>> for G in izip(*[chain(D.items(), repeat(None, N-1))]*N):

.... print G
....
(('a', 1), ('c', 3), ('b', 2))
(('e', 5), ('d', 4), ('g', 7))
(('f', 6), ('i', 9), ('h', )
(('j', 10), None, None)

I'd prefer the last sequence not return None
elements and instead just return (('j',10)), but this isn't a huge
deal.

This works and is clear, but it makes copies of items:

>>> ii = D.items()
>>> for i in range (0, len(ii), N):

.... print ii[i:i+N]
....
[('a', 1), ('c', 3), ('b', 2)]
[('e', 5), ('d', 4), ('g', 7)]
[('f', 6), ('i', 9), ('h', ]
[('j', 10)]

--
Noah
 
Reply With Quote
 
 
 
 
Paul Rubin
Guest
Posts: n/a
 
      03-14-2008
Noah <(E-Mail Removed)> writes:
> What is the fastest way to select N items at a time from a dictionary?
> I'm iterating over a dictionary of many thousands of items.
> I want to operate on only 100 items at a time.
> I want to avoid copying items using any sort of slicing.


I'd do something like (untested):

def groups(seq, n):
while True:
s = list(itertools.islice(seq, n))
if not s: return
yield s

items = d.iteritems()
for g in groups(items, 100):
operate_on (g)

> Does itertools copy items?


I don't understand this question.
 
Reply With Quote
 
 
 
 
attn.steven.kuo@gmail.com
Guest
Posts: n/a
 
      03-14-2008
On Mar 13, 6:34 pm, Noah <(E-Mail Removed)> wrote:
> What is the fastest way to select N items at a time from a dictionary?
> I'm iterating over a dictionary of many thousands of items.
> I want to operate on only 100 items at a time.
> I want to avoid copying items using any sort of slicing.
> Does itertools copy items?
>
> This works, but is ugly:
>
> >>> from itertools import *
> >>> D = {'a':1, 'b':2, 'c':3, 'd':4, 'e':5, 'f':6, 'g':7, 'h':8, 'i':9, 'j':10}
> >>> N = 3
> >>> for G in izip(*[chain(D.items(), repeat(None, N-1))]*N):

>
> ... print G
> ...
> (('a', 1), ('c', 3), ('b', 2))
> (('e', 5), ('d', 4), ('g', 7))
> (('f', 6), ('i', 9), ('h', )
> (('j', 10), None, None)
>
> I'd prefer the last sequence not return None
> elements and instead just return (('j',10)), but this isn't a huge
> deal.
>
> This works and is clear, but it makes copies of items:
>
> >>> ii = D.items()
> >>> for i in range (0, len(ii), N):

>
> ... print ii[i:i+N]
> ...
> [('a', 1), ('c', 3), ('b', 2)]
> [('e', 5), ('d', 4), ('g', 7)]
> [('f', 6), ('i', 9), ('h', ]
> [('j', 10)]
>



groupby?

import itertools

D = {'a':1, 'b':2, 'c':3, 'd':4, 'e':5, 'f':6, 'g':7, 'h':8, 'i':9,
'j':10}
N = 3

it = itertools.groupby(enumerate(D.items()), lambda t: int(t[0]/N))

for each in it:
print tuple(t[1] for t in each[1])

--
Hope this helps,
Steven
 
Reply With Quote
 
Arnaud Delobelle
Guest
Posts: n/a
 
      03-14-2008
On Mar 14, 1:34*am, Noah <(E-Mail Removed)> wrote:
> What is the fastest way to select N items at a time from a dictionary?
> I'm iterating over a dictionary of many thousands of items.
> I want to operate on only 100 items at a time.
> I want to avoid copying items using any sort of slicing.
> Does itertools copy items?
>
> This works, but is ugly:
>
> >>> from itertools import *
> >>> D = {'a':1, 'b':2, 'c':3, 'd':4, 'e':5, 'f':6, 'g':7, 'h':8, 'i':9, 'j':10}
> >>> N = 3
> >>> for G in izip(*[chain(D.items(), repeat(None, N-1))]*N):


This solution matches exactly the one proposed in itertools. The
following is an extract from http://docs.python.org/lib/itertools-functions.html.

Note, the left-to-right evaluation order of the iterables is
guaranteed. This makes possible an idiom for clustering a data series
into n-length groups using "izip(*[iter(s)]*n)". For data that doesn't
fit n-length groups exactly, the last tuple can be pre-padded with
fill values using "izip(*[chain(s, [None]*(n-1))]*n)".

--
Arnaud
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
dict.items() vs dict.iteritems and similar questions Drew Python 19 03-15-2007 09:23 PM
Newbie: create a clickable, grouped list/menu/whatever for items in an ecommerce site kbutterly ASP .Net 1 01-16-2007 01:57 PM
Displaying recordset with items grouped by year/month? Ken Fine ASP General 4 04-05-2004 08:03 AM
<logic:iterate /> iterate beyond items in the collection Gogo Java 1 09-04-2003 08:40 PM



Advertisments