Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Pack optimization

Reply
Thread Tools

Pack optimization

 
 
BDL
Guest
Posts: n/a
 
      10-15-2009
I have a large amount of binary data that needs to be pushed across
the network. It appears from profiling that the dominant time is
being taken up by packing the data. (50%) Here is a CME that shows
the problem.

from numpy import random
from struct import pack
import time

lenstim = 10084200
sigma = 0.1
stim = random.normal(0., sigma, lenstim) # 10084200 gaussian random
numbers (doubles)

fmt = '!h'+str(stim.size)+'d' # makes fmt = '!h10084200d'
cmd = 4

startTime = time.time()
packdat = pack( fmt, cmd, *stim )
elapsed = time.time() - startTime
print "Time to pack the command and data %.6f seconds " % elapsed

Is there a faster method to do this? Is it possible to use array?
Any suggestions would be appreciated.
 
Reply With Quote
 
 
 
 
Paul Rubin
Guest
Posts: n/a
 
      10-15-2009
BDL <(E-Mail Removed)> writes:
> Is there a faster method to do this? Is it possible to use array?


array.tostring is the first thing I'd think of.
 
Reply With Quote
 
 
 
 
Robert Kern
Guest
Posts: n/a
 
      10-15-2009
On 2009-10-15 13:57 PM, BDL wrote:
> I have a large amount of binary data that needs to be pushed across
> the network. It appears from profiling that the dominant time is
> being taken up by packing the data. (50%) Here is a CME that shows
> the problem.
>
> from numpy import random
> from struct import pack
> import time
>
> lenstim = 10084200
> sigma = 0.1
> stim = random.normal(0., sigma, lenstim) # 10084200 gaussian random
> numbers (doubles)
>
> fmt = '!h'+str(stim.size)+'d' # makes fmt = '!h10084200d'
> cmd = 4
>
> startTime = time.time()
> packdat = pack( fmt, cmd, *stim )
> elapsed = time.time() - startTime
> print "Time to pack the command and data %.6f seconds " % elapsed
>
> Is there a faster method to do this? Is it possible to use array?
> Any suggestions would be appreciated.


If you're already using numpy, use its routines to convert to string
representations (like stim.tostring(), but there are better alternatives like
the NPY file format). Don't use struct.pack() for large homogeneous data.

If you have further numpy questions, you should ask them on the numpy mailing list:

http://www.scipy.org/Mailing_Lists

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Zero Optimization and Sign Optimization??? Ravikiran C Programming 22 11-24-2008 03:19 AM
Buy 1 pack or 3 pack? (XP Pro x64) Rich Olver Windows 64bit 11 11-25-2006 11:33 PM
Excellent discount software packs - ImTOO Ripper Pack Platinum and ImTOO Ripper Pack Gold zhangelf01@gmail.com DVD Video 6 09-17-2006 03:27 AM
Pack parent control in child using control.pack(in_= syntax? Tim Jones Python 0 01-31-2004 10:22 PM
Poor Mans NIMH Battery Pack and SLA Battery Pack ajacobs2 Digital Photography 0 08-19-2003 12:42 PM



Advertisments