Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Populating a dictionary, fast

Reply
Thread Tools

Re: Populating a dictionary, fast

 
 
Michael Bacarella
Guest
Posts: n/a
 
      11-11-2007
> > This would seem to implicate the line id2name[id] = name as being
excruciatingly slow.
>
> As others have pointed out there is no way that this takes 45
> minutes.Must be something with your system or setup.
>
> A functionally equivalent code for me runs in about 49 seconds!
> (it ends up using about twice as much ram as the data on disk)


You can download the list of keys from here, it's 43M gzipped:
http://www.sendspace.com/file/9530i7

and see it take about 45 minutes with this:

$ cat cache-keys.py
#!/usr/bin/python
v = {}
for line in open('keys.txt'):
v[long(line.strip())] = True



 
Reply With Quote
 
 
 
 
Marc 'BlackJack' Rintsch
Guest
Posts: n/a
 
      11-11-2007
On Sun, 11 Nov 2007 08:51:37 -0800, Michael Bacarella wrote:

>> As others have pointed out there is no way that this takes 45
>> minutes.Must be something with your system or setup.
>>
>> A functionally equivalent code for me runs in about 49 seconds!
>> (it ends up using about twice as much ram as the data on disk)

>
> You can download the list of keys from here, it's 43M gzipped:
> http://www.sendspace.com/file/9530i7
>
> and see it take about 45 minutes with this:
>
> $ cat cache-keys.py
> #!/usr/bin/python
> v = {}
> for line in open('keys.txt'):
> v[long(line.strip())] = True


Takes about 40 seconds here.

bj@s8n:~$ time python test.py

real 0m38.758s
user 0m25.290s
sys 0m1.580s

Ciao,
Marc 'BlackJack' Rintsch
 
Reply With Quote
 
 
 
 
Istvan Albert
Guest
Posts: n/a
 
      11-12-2007
On Nov 11, 11:51 am, Michael Bacarella <(E-Mail Removed)> wrote:

> and see it take about 45 minutes with this:
>
> $ cat cache-keys.py
> #!/usr/bin/python
> v = {}
> for line in open('keys.txt'):
> v[long(line.strip())] = True


On my system (windows vista) your code (using your data) runs in:

36 seconds with python 2.4
25 seconds with python 2.5
39 seconds with python 3000


i.


 
Reply With Quote
 
Michael Bacarella
Guest
Posts: n/a
 
      11-12-2007
> > and see it take about 45 minutes with this:
> >
> > $ cat cache-keys.py
> > #!/usr/bin/python
> > v = {}
> > for line in open('keys.txt'):
> > v[long(line.strip())] = True

>
> On my system (windows vista) your code (using your data) runs in:
>
> 36 seconds with python 2.4
> 25 seconds with python 2.5
> 39 seconds with python 3000


*wow*

The win32 Python or the cygwin Python?

What CPU architecture?


 
Reply With Quote
 
Istvan Albert
Guest
Posts: n/a
 
      11-12-2007
On Nov 12, 12:39 pm, "Michael Bacarella" <(E-Mail Removed)> wrote:

> The win32 Python or the cygwin Python?
>
> What CPU architecture?


it is the win32 version, a dual core laptop with T5220 Core 2

 
Reply With Quote
 
MrJean1
Guest
Posts: n/a
 
      11-12-2007
On MacOS** your version

#!/usr/bin/python
v = {}
for line in open('keys.txt'):
v[long(line.strip())] = True

using these <http://www.sendspace.com/file/9530i7> keys takes

24.9 secs with ActivePython 2.5.1
and
212.3 secs with Apple's Python 2.3.5.


However, this version

#!/usr/bin/python
v = {}
for line in open('keys.txt'):
v[line] = True

takes
16.5 secs with ActivePython 2.5.1
and
18.7 secs with Apple's Python 2.3.5.


/Jean Brouwers

**) MacOS X 10.4.10 on Mini Mac, 1.83 GHz Intel Core Duo, 2 GB RAM.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: Populating a dictionary, fast Michael Bacarella Python 16 11-21-2007 08:17 PM
Re: Populating a dictionary, fast Michael Bacarella Python 2 11-12-2007 03:41 PM
Re: Populating a dictionary, fast Michael Bacarella Python 6 11-12-2007 10:37 AM
Re: Populating a dictionary, fast Michael Bacarella Python 3 11-11-2007 10:23 PM
Populating a dictionary, fast Michael Bacarella Python 4 11-11-2007 04:11 PM



Advertisments