Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > profiling and performance of shelves

Reply
Thread Tools

profiling and performance of shelves

 
 
Eric S. Johansson
Guest
Posts: n/a
 
      06-22-2004
was profiling some of my code trying to figure out just why it was running slow and I discovered that shelves containing dictionaries were significantly slower than those containing simple tuples. for example, on a Pentium III/600 MHz machine the code below ran in 1.12 seconds but when I substituted a simple tuple for the shelf, it ran significantly faster (0.13 seconds). I'm confused as to why shelving dictionaries (all elements textual) should take so much longer is the pickel process that complex?.

----- test framework -----
#!/usr/bin/python

import sys

development_path = "/usr/local/camram/modules"
sys.path.insert(1,development_path)

import dbm_utils
import camram_utils
import configuration

#set the local context
config_data = configuration.configuration("esj")

def main():

# wrapper for shelf which also includes locking, file creation etc.
# returns object derived from type shelve.
spamtrap_cache = camram_utils.message_cache( configuration_data = config_data)

cnt =0
x = 0

# test code. Don't use without adding filename to message_cache
## while x<500:
## spamtrap_cache[str(x)]=(x,"const string")
## x=x+1

for i in spamtrap_cache.keys():
token = spamtrap_cache[i]
cnt=cnt+1

print cnt
print token

import profile
profile.run('main()', '/tmp/speed')

import pstats

stats = pstats.Stats('/tmp/speed')
stats.strip_dirs().sort_stats('cumulative').print_ stats()
stats.print_callees()

---- profile output

Tue Jun 22 14:25:48 2004 /tmp/speed

12479 function calls in 1.120 CPU seconds

Ordered by: cumulative time

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.120 1.120 <string>:1(?)
1 0.000 0.000 1.120 1.120 profile:0(main())
1 0.030 0.030 1.120 1.120 speed.py:13(main)
566 0.960 0.002 1.080 0.002 shelve.py:69(__getitem__)
11887 0.120 0.000 0.120 0.000 <string>:0(?)
1 0.010 0.010 0.010 0.010 shelve.py:55(keys)
2 0.000 0.000 0.000 0.000 posixpath.py:362(normpath)
0 0.000 0.000 profile:0(profiler)
1 0.000 0.000 0.000 0.000 shelve.py:82(close)
1 0.000 0.000 0.000 0.000 whichdb.py:5(whichdb)
1 0.000 0.000 0.000 0.000 anydbm.py:69(open)
2 0.000 0.000 0.000 0.000 configuration.py:294(__getitem__)
2 0.000 0.000 0.000 0.000 configuration.py:355(path_process)
1 0.000 0.000 0.000 0.000 dbm_utils.py:53(bias_anydbm)
1 0.000 0.000 0.000 0.000 shelve.py:52(__init__)
1 0.000 0.000 0.000 0.000 dbm_utils.py:359(__init__)
2 0.000 0.000 0.000 0.000 configuration.py:80(base_pathname)
2 0.000 0.000 0.000 0.000 posixpath.py:299(expanduser)
1 0.000 0.000 0.000 0.000 portalocker.py:68(lock)
1 0.000 0.000 0.000 0.000 shelve.py:89(__del__)
1 0.000 0.000 0.000 0.000 camram_utils.py:329(close)
2 0.000 0.000 0.000 0.000 posixpath.py:330(expandvars)
1 0.000 0.000 0.000 0.000 camram_utils.py:314(__init__)



Ordered by: cumulative time

Function called...
<string>:1(?) shelve.py:89(__del__)(1) 0.000
speed.py:13(main)(1) 1.120
profile:0(main()) <string>:1(?)(1) 1.120
speed.py:13(main) camram_utils.py:314(__init__)(1) 0.000
shelve.py:55(keys)(1) 0.010
shelve.py:69(__getitem__)(566) 1.080
shelve.py:69(__getitem__) <string>:0(?)(11887) 0.120
<string>:0(?) --
shelve.py:55(keys) --
posixpath.py:362(normpath) --
profile:0(profiler) profile:0(main())(1) 1.120
shelve.py:82(close) --
whichdb.py:5(whichdb) --
anydbm.py:69(open) whichdb.py:5(whichdb)(1) 0.000
configuration.py:294(__getitem__) configuration.py:355(path_process)(2) 0.000
configuration.py:355(path_process) configuration.py:80(base_pathname)(2) 0.000
dbm_utils.py:53(bias_anydbm) --
shelve.py:52(__init__) --
dbm_utils.py:359(__init__) anydbm.py:69(open)(1) 0.000
dbm_utils.py:53(bias_anydbm)(1) 0.000
shelve.py:52(__init__)(1) 0.000
configuration.py:80(base_pathname) posixpath.py:299(expanduser)(2) 0.000
posixpath.py:330(expandvars)(2) 0.000
posixpath.py:362(normpath)(2) 0.000
posixpath.py:299(expanduser) --
portalocker.py:68(lock) --
shelve.py:89(__del__) camram_utils.py:329(close)(1) 0.000
camram_utils.py:329(close) shelve.py:82(close)(1) 0.000
posixpath.py:330(expandvars) --
camram_utils.py:314(__init__) configuration.py:294(__getitem__)(2) 0.000
dbm_utils.py:359(__init__)(1) 0.000
portalocker.py:68(lock)(1) 0.000






 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
A way to read shelves quickly? zmason@gmail.com Python 1 04-06-2006 03:00 AM
Barnes & Noble Removing All Microsoft Related Magazines From Shelves clintonG ASP .Net 9 03-01-2006 05:27 AM
Ikea Billy shelves as storage w.beckley@gmail.com DVD Video 5 06-05-2005 04:18 AM
High Definition DVD on shelves? Marshall Dudley DVD Video 15 12-01-2004 04:41 AM
Re: profiling and performance of shelves Eric S. Johansson Python 0 06-25-2004 12:55 PM



Advertisments