Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Printing to file, how do I do it efficiently?

Reply
Thread Tools

Printing to file, how do I do it efficiently?

 
 
Cameron Walsh
Guest
Posts: n/a
 
      11-10-2006
Hi all,

I have a numpy.array of 89x512x512 uint8's, set up with code like this:


data=numpy.array([],dtype="uint8")
data.resize((89,512,512))
# Data filled in about 4 seconds from 89 image slices
<snip lots of processing code>


I first tried writing this data to a binary raw format (for use in a
program called Drishti) as follows:

# The slow bit:
volumeFile=file("/tmp/test.raw","wb")
for z in xrange(Z):
for y in xrange(Y):
for x in xrange(X):
volumeFile.write("%c" %(data[z,y,x]))
volumeFile.close()

That took about 39 seconds.

My second attempt was as follows:
volumeFile = open("/tmp/test2.raw","wb")
data.resize((X*Y*Z)) # Flatten the array
for i in data:
volumeFile.write("%c" %i)
data.resize((Z,Y,X))
volumeFile.close()

This took 32 seconds. (For those of you unfamiliar with numpy, the
data.resize() operations take negligible amounts of time, all it does is
allow the data to be accessed differently.)

I'm guessing that the slow part is the fact that I am converting the
data to character format and writing it one character at a time. What
is a better way of doing this, or where should I look to find a better way?


Thanks,

Cameron.
 
Reply With Quote
 
 
 
 
Robert Kern
Guest
Posts: n/a
 
      11-10-2006
Cameron Walsh wrote:
> Hi all,
>
> I have a numpy.array of 89x512x512 uint8's, set up with code like this:


numpy questions are best asked on the numpy list, not here.

http://www.scipy.org/Mailing_Lists

> data=numpy.array([],dtype="uint8")
> data.resize((89,512,512))


You might want to look at using numpy.empty() here, instead.

> # Data filled in about 4 seconds from 89 image slices
> <snip lots of processing code>
>
>
> I first tried writing this data to a binary raw format (for use in a
> program called Drishti) as follows:
>
> # The slow bit:
> volumeFile=file("/tmp/test.raw","wb")
> for z in xrange(Z):
> for y in xrange(Y):
> for x in xrange(X):
> volumeFile.write("%c" %(data[z,y,x]))
> volumeFile.close()
>
> That took about 39 seconds.
>
> My second attempt was as follows:
> volumeFile = open("/tmp/test2.raw","wb")
> data.resize((X*Y*Z)) # Flatten the array
> for i in data:
> volumeFile.write("%c" %i)
> data.resize((Z,Y,X))
> volumeFile.close()
>
> This took 32 seconds. (For those of you unfamiliar with numpy, the
> data.resize() operations take negligible amounts of time, all it does is
> allow the data to be accessed differently.)


No, if the total size is different, it will also copy the array. Use .reshape()
if you want to simply alter the shape, not the total number of elements.

> I'm guessing that the slow part is the fact that I am converting the
> data to character format and writing it one character at a time. What
> is a better way of doing this, or where should I look to find a better way?


data.tostring()

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 
Reply With Quote
 
 
 
 
Cameron Walsh
Guest
Posts: n/a
 
      11-10-2006
Robert Kern wrote:
> Cameron Walsh wrote:
>> Hi all,
>>
>> I have a numpy.array of 89x512x512 uint8's, set up with code like this:

>
> numpy questions are best asked on the numpy list, not here.


At first I thought it was a generic python question, since it had more
to do with writing array data to file rather than the specific format of
the array data.

>
>> data=numpy.array([],dtype="uint8")
>> data.resize((89,512,512))

>
> You might want to look at using numpy.empty() here, instead.
>


Thanks!

[...]
>> I'm guessing that the slow part is the fact that I am converting the
>> data to character format and writing it one character at a time. What
>> is a better way of doing this, or where should I look to find a better way?

>
> data.tostring()
>


And here I see I was wrong, it was a numpy question. I assumed the
tostring() method would produce the same output as printing the array to
the screen by just calling "data". But of course, that would be the job
of the __repr__() method.

It is now ridiculously fast (<1second). Thank you for your help.

Cameron.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
brochure printing,online yearbook,printing,books printing,publishing elie Computer Support 0 08-21-2007 05:52 AM
brochure printing,online yearbook,printing,books printing,publishing elie Computer Support 0 08-21-2007 05:50 AM
brochure printing,online yearbook,printing,books printing,publishing elie Computer Support 0 08-21-2007 05:28 AM
brochure printing,online yearbook,printing,books printing,publishing elie Computer Support 0 08-18-2007 10:11 AM



Advertisments