Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Best way to add a "position" value to each item in a list

Reply
Thread Tools

Best way to add a "position" value to each item in a list

 
 
Sean
Guest
Posts: n/a
 
      07-09-2009
I have a huge list, 10,000,000+ items. Each item is a dictionary with
fields used to sort the list. When I have completed sorting I want to
grab a page of items, say 1,000 of them which I do easily by using
list_data[x+1000]

Now I want to add an additional key/value pair to each dictionary in
the list, incrementing them by 1 each time. So, if I grabbed page 2
of the list I would get:

[{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
1002}, ...]

Any way to do that with list comprehension? Any other good way to do
it besides iterating over the list?

Thanks
 
Reply With Quote
 
 
 
 
Sean
Guest
Posts: n/a
 
      07-09-2009
On Jul 9, 1:16*pm, Sean <(E-Mail Removed)> wrote:
> I have a huge list, 10,000,000+ items. *Each item is a dictionary with
> fields used to sort the list. *When I have completed sorting I want to
> grab a page of items, say 1,000 of them which I do easily by using
> list_data[x+1000]
>
> Now I want to add an additional key/value pair to each dictionary in
> the list, incrementing them by 1 each time. *So, if I grabbed page 2
> of the list I would get:
>
> [{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
> 1002}, ...]
>
> Any way to do that with list comprehension? *Any other good way to do
> it besides iterating over the list?
>
> Thanks


I was able to do this by doing the following:
page_data = [[start_position + 1 + n, x] for n, x in enumerate
(page_data)]

However, this creates a list of lists, each containing an int and a
dictionary. I wanted to add it directly to the dictionary. I can add
the key position to the dictionary when the data is created, but how
would I assign the value in a list comprehension?

 
Reply With Quote
 
 
 
 
Pablo Torres N.
Guest
Posts: n/a
 
      07-09-2009
Howdy,

On Thu, Jul 9, 2009 at 15:16, Sean<(E-Mail Removed)> wrote:
> I have a huge list, 10,000,000+ items. *Each item is a dictionary with
> fields used to sort the list. *When I have completed sorting I want to
> grab a page of items, say 1,000 of them which I do easily by using
> list_data[x+1000]
>
> Now I want to add an additional key/value pair to each dictionary in
> the list, incrementing them by 1 each time. *So, if I grabbed page 2
> of the list I would get:
>
> [{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
> 1002}, ...]
>


I don't get it, what do you increment by one, the value of a given key
or the number of key/value pairs? Also, if the 'position' key is the
index of the item in the list, then I don't understand what you mean
by 'page'. Could you tell us about the structure of these
dictionaries?

> Any way to do that with list comprehension? *Any other good way to do
> it besides iterating over the list?
>
> Thanks
> --
> http://mail.python.org/mailman/listinfo/python-list
>




--
Pablo Torres N.
 
Reply With Quote
 
Jerry Hill
Guest
Posts: n/a
 
      07-09-2009
On Thu, Jul 9, 2009 at 4:16 PM, Sean<(E-Mail Removed)> wrote:
> I have a huge list, 10,000,000+ items. *Each item is a dictionary with
> fields used to sort the list. *When I have completed sorting I want to
> grab a page of items, say 1,000 of them which I do easily by using
> list_data[x+1000]
>
> Now I want to add an additional key/value pair to each dictionary in
> the list, incrementing them by 1 each time. *So, if I grabbed page 2
> of the list I would get:
>
> [{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
> 1002}, ...]
>
> Any way to do that with list comprehension? *Any other good way to do
> it besides iterating over the list?


Normally you wouldn't mutate the items in a list with a list
comprehension. Instead, you would use a for loop, like this:

for idx,item in enumerate(my_list_of_dicts):
item['position'] = idx

Is there a particular reason you want to do this with a list
comprehension? Using a list comp means you're going to create an
extra copy of your 10 million item list, just so you can add a key to
each member, then (probably) throw away the original list. It doesn't
seem like the right tool for the job here.

--
Jerry
 
Reply With Quote
 
Sean
Guest
Posts: n/a
 
      07-09-2009
On Jul 9, 1:16*pm, Sean <(E-Mail Removed)> wrote:
> I have a huge list, 10,000,000+ items. *Each item is a dictionary with
> fields used to sort the list. *When I have completed sorting I want to
> grab a page of items, say 1,000 of them which I do easily by using
> list_data[x+1000]
>
> Now I want to add an additional key/value pair to each dictionary in
> the list, incrementing them by 1 each time. *So, if I grabbed page 2
> of the list I would get:
>
> [{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
> 1002}, ...]
>
> Any way to do that with list comprehension? *Any other good way to do
> it besides iterating over the list?
>
> Thanks


I was able to do this by doing the following:
page_data = [[start_position + 1 + n, x] for n, x in enumerate
(page_data)]

However, this creates a list of lists, each containing an int and a
dictionary. I wanted to add it directly to the dictionary. I can add
the key position to the dictionary when the data is created, but how
would I assign the value in a list comprehension?

 
Reply With Quote
 
Carl Banks
Guest
Posts: n/a
 
      07-09-2009
On Jul 9, 1:16*pm, Sean <(E-Mail Removed)> wrote:
> I have a huge list, 10,000,000+ items. *Each item is a dictionary with
> fields used to sort the list. *When I have completed sorting I want to
> grab a page of items, say 1,000 of them which I do easily by using
> list_data[x+1000]
>
> Now I want to add an additional key/value pair to each dictionary in
> the list, incrementing them by 1 each time. *So, if I grabbed page 2
> of the list I would get:
>
> [{'a':'a', 'b':'b', 'position':1001}, {'c':'c', 'd':'d', 'position':
> 1002}, ...]
>
> Any way to do that with list comprehension? *Any other good way to do
> it besides iterating over the list?



Not really, but if you want to avoid the delay in setting the
dictionary elements (I'm guessing that is why you don't want to
iterate over the list--and parenthetically it's not iterating over the
list but adding dictionary items that is driving the time, and that
cost is unavoidable), you should consider adding the position elements
on demand. That is, instead of running a big loop once to add
position to all ten million elements, just run the loop on individual
pages.

def get_page(start,end):
page = list_data[start:end]
for i,item in enumerate(page):
page['position'] = start+i
return page


That might not work depending on what you are doing but you should
consider it.


Carl Banks
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Help! acts_as_nested_set Administration (add new item, delete item) Namor Ruby 0 01-19-2006 08:26 AM
why writing list to file puts each item from list on seperate line? homepricemaps@gmail.com Python 5 12-31-2005 05:21 AM
why writing list to file puts each item from list on seperate line? homepricemaps@gmail.com Python 0 12-31-2005 04:22 AM
Change style of a single row of the item list of datagrid, based on a field value of current item... QUASAR ASP .Net Datagrid Control 6 01-17-2004 07:46 PM
Display Related Item or Add Item ssoss ASP .Net 2 09-18-2003 11:35 PM



Advertisments