Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Fast forward-backward (write-read)

Reply
Thread Tools

Re: Fast forward-backward (write-read)

 
 
David Hutto
Guest
Posts: n/a
 
      10-24-2012
On Tue, Oct 23, 2012 at 8:06 PM, Oscar Benjamin
<(E-Mail Removed)> wrote:
> On 23 October 2012 15:31, Virgil Stokes <(E-Mail Removed)> wrote:
>> I am working with some rather large data files (>100GB) that contain time
>> series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
>> format. I perform various types of processing on these data (e.g. moving
>> median, moving average, and Kalman-filter, Kalman-smoother) in a sequential
>> manner and only a small number of these data need be stored in RAM when
>> being processed. When performing Kalman-filtering (forward in time pass, k =
>> 0,1,...,N) I need to save to an external file several variables (e.g. 11*32
>> bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother
>> (backward in time pass, k = N,N-1,...,0). Thus, I will need to input these
>> variables saved to an external file from the forward pass, in reverse order
>> --- from last written to first written.
>>
>> Finally, to my question --- What is a fast way to write these variables to
>> an external file and then read them in backwards?

>
> You mentioned elsewhere that you are using numpy. I'll assume that the
> data you want to read/write are numpy arrays.


If that is the case always timeit. The following is an example of 3
functions, with repetitions of time that give an average:

import timeit
#3 dimensional matrix
x_dim = -1
y_dim = -1
z_dim = -1
s = """\

x_dim = -1
y_dim = -1
z_dim = -1
dict_1 = {}

for i in xrange(0,6):
x_dim = 1
y_dim = 1
z_dim = 1
dict_1['%s' % (i) ] = ['x = %i' % (x_dim), 'y = %i' % (y_dim), 'z =
%i' % (z_dim)]

"""

t = """\
import numpy
numpy.array([[ 1., 0., 0.],
[ 0., 1., 2.]])
"""

u = """\
list_count = 0
an_array = []
for i in range(0,10):

if list_count > 3:
break

if i % 3 != 0:
an_array.append(i)

if i % 3 == 0:
list_count += 1

"""
print timeit.timeit(stmt=s, number=100000)
print timeit.timeit(stmt=t, number=100000)
print timeit.timeit(stmt=u, number=100000)


--
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
[ANN] macstl 0.2 -- portable SIMD toolkit, fast valarray transcendentals, fast Mach vectors glenlow@pixelglow.com C++ 0 02-02-2005 12:32 PM
More memory: How fast is fast rfdjr1@optonline.net Computer Support 5 05-19-2004 05:45 PM
Canon S30 Fast shutter mode... Why so fast? mark popp Digital Photography 1 02-08-2004 10:07 PM
I NEED HELP FAST!!!!! REAL FAST!!!!! R. Jizzle MCSE 3 09-29-2003 08:51 PM
Super-fast AA Chargers: Anything as fast as the 15 minute Rayovac? David Chien Digital Photography 4 08-30-2003 07:49 PM



Advertisments