Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Anyone happen to have optimization hints for this loop?

Reply
Thread Tools

Anyone happen to have optimization hints for this loop?

 
 
dp_pearce
Guest
Posts: n/a
 
      07-09-2008
I have some code that takes data from an Access database and processes
it into text files for another application. At the moment, I am using
a number of loops that are pretty slow. I am not a hugely experienced
python user so I would like to know if I am doing anything
particularly wrong or that can be hugely improved through the use of
another method.

Currently, all of the values that are to be written to file are pulled
from the database and into a list called "domainVa". These values
represent 3D data and need to be written to text files using line
breaks to seperate 'layers'. I am currently looping through the list
and appending a string, which I then write to file. This list can
regularly contain upwards of half a million values...

count = 0
dmntString = ""
for z in range(0, Z):
for y in range(0, Y):
for x in range(0, X):
fraction = domainVa[count]
dmntString += " "
dmntString += fraction
count = count + 1
dmntString += "\n"
dmntString += "\n"
dmntString += "\n***\n

dmntFile = open(dmntFilename, 'wt')
dmntFile.write(dmntString)
dmntFile.close()

I have found that it is currently taking ~3 seconds to build the
string but ~1 second to write the string to file, which seems wrong (I
would normally guess the CPU/Memory would out perform disc writing
speeds).

Can anyone see a way of speeding this loop up? Perhaps by changing the
data format? Is it wrong to append a string and write once, or should
hold a file open and write at each instance?

Thank you in advance for your time,

Dan
 
Reply With Quote
 
 
 
 
Diez B. Roggisch
Guest
Posts: n/a
 
      07-09-2008
dp_pearce wrote:

> I have some code that takes data from an Access database and processes
> it into text files for another application. At the moment, I am using
> a number of loops that are pretty slow. I am not a hugely experienced
> python user so I would like to know if I am doing anything
> particularly wrong or that can be hugely improved through the use of
> another method.
>
> Currently, all of the values that are to be written to file are pulled
> from the database and into a list called "domainVa". These values
> represent 3D data and need to be written to text files using line
> breaks to seperate 'layers'. I am currently looping through the list
> and appending a string, which I then write to file. This list can
> regularly contain upwards of half a million values...
>
> count = 0
> dmntString = ""
> for z in range(0, Z):
> for y in range(0, Y):
> for x in range(0, X):
> fraction = domainVa[count]
> dmntString += " "
> dmntString += fraction
> count = count + 1
> dmntString += "\n"
> dmntString += "\n"
> dmntString += "\n***\n
>
> dmntFile = open(dmntFilename, 'wt')
> dmntFile.write(dmntString)
> dmntFile.close()
>
> I have found that it is currently taking ~3 seconds to build the
> string but ~1 second to write the string to file, which seems wrong (I
> would normally guess the CPU/Memory would out perform disc writing
> speeds).
>
> Can anyone see a way of speeding this loop up? Perhaps by changing the
> data format? Is it wrong to append a string and write once, or should
> hold a file open and write at each instance?


Don't use in-place adding to concatenate strings. It might lead to
quadaratic behavior.

Use the "".join()-idiom instead:

dmntStrings = []
.....
dmntStrings.append("\n")
.....
dmntFile.write("".join(dmntStrings))

Diez
 
Reply With Quote
 
 
 
 
Terry Reedy
Guest
Posts: n/a
 
      07-09-2008


Diez B. Roggisch wrote:
> dp_pearce wrote:




>>
>> count = 0
>> dmntString = ""
>> for z in range(0, Z):
>> for y in range(0, Y):
>> for x in range(0, X):
>> fraction = domainVa[count]
>> dmntString += " "
>> dmntString += fraction


Minor point, just construct " "+domain[count] all at once

>> count = count + 1
>> dmntString += "\n"
>> dmntString += "\n"
>> dmntString += "\n***\n
>>
>> dmntFile = open(dmntFilename, 'wt')
>> dmntFile.write(dmntString)
>> dmntFile.close()
>>
>> I have found that it is currently taking ~3 seconds to build the
>> string but ~1 second to write the string to file, which seems wrong (I
>> would normally guess the CPU/Memory would out perform disc writing
>> speeds).
>>
>> Can anyone see a way of speeding this loop up? Perhaps by changing the
>> data format? Is it wrong to append a string and write once, or should
>> hold a file open and write at each instance?

>
> Don't use in-place adding to concatenate strings. It might lead to
> quadaratic behavior.


Semantically, repeated extension of an immutable is inherently
quadratic. And it is for strings in Python until 2.6 or possibly 2.5
(not sure), when more sophisticated code was added because people kept
falling into this trap. But since the more sophisticated code 'cheats'
by mutating the immutable (with an algorithm similar to list.extend),I
am pretty sure it will only be used when there is only one reference to
the string and the extension is with +=, so that the extension can
reliably be done in-place without changing semantics. Thus, Python
programmers should still learn the following.

> Use the "".join()-idiom instead:


(Or upgrade)
>
> dmntStrings = []
> ....
> dmntStrings.append("\n")
> ....
> dmntFile.write("".join(dmntStrings))


Note that the minor point above will cut the number of things to be
joined nearly in half.

tjr

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
hints for find-as-you-type hints cbossens73@yahoo.fr Java 2 04-08-2009 11:08 AM
Zero Optimization and Sign Optimization??? Ravikiran C Programming 22 11-24-2008 03:19 AM
Anyone happen to have optimization hints for this loop? dp_pearce Python 5 07-15-2008 02:10 PM
HTML table border - has anyone seen this happen before? throat.wobbler.mangrove@gmail.com HTML 8 11-24-2006 09:45 AM
Help and optimization hints, anyone? Kim Petersen Python 4 01-23-2004 04:39 PM



Advertisments