Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > quickly looping over a 2D array?

Reply
Thread Tools

quickly looping over a 2D array?

 
 
Martin
Guest
Posts: n/a
 
      07-27-2009
Hi,

I am new to python and I was wondering if there was a way to speed up
the way I index 2D arrays when I need to check two arrays
simultaneously? My current implementations is (using numpy) something
like the following...

for i in range(numrows):
for j in range(numcols):

if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
j] * some_other_value
array_1[i, j] = some_new_value

Many Thanks,

Martin
 
Reply With Quote
 
 
 
 
Peter Otten
Guest
Posts: n/a
 
      07-27-2009
Martin wrote:

> I am new to python and I was wondering if there was a way to speed up
> the way I index 2D arrays when I need to check two arrays
> simultaneously? My current implementations is (using numpy) something
> like the following...
>
> for i in range(numrows):
> for j in range(numcols):
>
> if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
> j] * some_other_value
> array_1[i, j] = some_new_value


array_1[(array_1 == some_value) | (array_2 >= array_1 * some_other_value)] =
some_new_value

maybe?

 
Reply With Quote
 
 
 
 
Martin
Guest
Posts: n/a
 
      07-27-2009
On Jul 27, 12:42 pm, Peter Otten <(E-Mail Removed)> wrote:
> Martin wrote:
> > I am new to python and I was wondering if there was a way to speed up
> > the way I index 2D arrays when I need to check two arrays
> > simultaneously? My current implementations is (using numpy) something
> > like the following...

>
> > for i in range(numrows):
> > for j in range(numcols):

>
> > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
> > j] * some_other_value
> > array_1[i, j] = some_new_value

>
> array_1[(array_1 == some_value) | (array_2 >= array_1 * some_other_value)] =
> some_new_value
>
> maybe?


So I tried...

band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

which led to

ValueError: The truth value of an array with more than one element is
ambiguous. Use a.any() or a.all()

so not sure that works?
 
Reply With Quote
 
Peter Otten
Guest
Posts: n/a
 
      07-27-2009
Martin wrote:

> On Jul 27, 12:42 pm, Peter Otten <(E-Mail Removed)> wrote:
>> Martin wrote:
>> > I am new to python and I was wondering if there was a way to speed up
>> > the way I index 2D arrays when I need to check two arrays
>> > simultaneously? My current implementations is (using numpy) something
>> > like the following...

>>
>> > for i in range(numrows):
>> > for j in range(numcols):

>>
>> > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
>> > j] * some_other_value
>> > array_1[i, j] = some_new_value

>>
>> array_1[(array_1 == some_value) | (array_2 >= array_1 *
>> some_other_value)] = some_new_value
>>
>> maybe?

>
> So I tried...
>
> band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0
>
> which led to
>
> ValueError: The truth value of an array with more than one element is
> ambiguous. Use a.any() or a.all()
>
> so not sure that works?


Copy and paste -- or replace "or" with "|".

 
Reply With Quote
 
Martin
Guest
Posts: n/a
 
      07-27-2009
On Jul 27, 1:46 pm, Peter Otten <(E-Mail Removed)> wrote:
> Martin wrote:
> > On Jul 27, 12:42 pm, Peter Otten <(E-Mail Removed)> wrote:
> >> Martin wrote:
> >> > I am new to python and I was wondering if there was a way to speed up
> >> > the way I index 2D arrays when I need to check two arrays
> >> > simultaneously? My current implementations is (using numpy) something
> >> > like the following...

>
> >> > for i in range(numrows):
> >> > for j in range(numcols):

>
> >> > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
> >> > j] * some_other_value
> >> > array_1[i, j] = some_new_value

>
> >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
> >> some_other_value)] = some_new_value

>
> >> maybe?

>
> > So I tried...

>
> > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

>
> > which led to

>
> > ValueError: The truth value of an array with more than one element is
> > ambiguous. Use a.any() or a.all()

>
> > so not sure that works?

>
> Copy and paste -- or replace "or" with "|".


apologies - I mistook that for a type for "or"

I now get the following error...

ValueError: shape mismatch: objects cannot be broadcast to a single
shape
 
Reply With Quote
 
Peter Otten
Guest
Posts: n/a
 
      07-27-2009
Martin wrote:

> On Jul 27, 1:46 pm, Peter Otten <(E-Mail Removed)> wrote:
>> Martin wrote:
>> > On Jul 27, 12:42 pm, Peter Otten <(E-Mail Removed)> wrote:
>> >> Martin wrote:
>> >> > I am new to python and I was wondering if there was a way to speed
>> >> > up the way I index 2D arrays when I need to check two arrays
>> >> > simultaneously? My current implementations is (using numpy)
>> >> > something like the following...

>>
>> >> > for i in range(numrows):
>> >> > for j in range(numcols):

>>
>> >> > if array_1[i, j] == some_value or array_2[i, j] >=
>> >> > array_1[i,
>> >> > j] * some_other_value
>> >> > array_1[i, j] = some_new_value

>>
>> >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
>> >> some_other_value)] = some_new_value

>>
>> >> maybe?

>>
>> > So I tried...

>>
>> > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

>>
>> > which led to

>>
>> > ValueError: The truth value of an array with more than one element is
>> > ambiguous. Use a.any() or a.all()

>>
>> > so not sure that works?

>>
>> Copy and paste -- or replace "or" with "|".

>
> apologies - I mistook that for a type for "or"
>
> I now get the following error...
>
> ValueError: shape mismatch: objects cannot be broadcast to a single
> shape


It seems array_1 and array_2 have a -- dada! -- different shape.
Assuming array_1 is the smaller one:

numrows, numcols = array_1.shape
array_1[(array_1 == some_value) | (array_2[:numrows,:numcols] >= array_1 *
some_other_value)] = some_new_value

There may be other options, but I'm not a numpy expert.

Peter

 
Reply With Quote
 
Martin
Guest
Posts: n/a
 
      07-27-2009
On Jul 27, 2:17 pm, Peter Otten <(E-Mail Removed)> wrote:
> Martin wrote:
> > On Jul 27, 1:46 pm, Peter Otten <(E-Mail Removed)> wrote:
> >> Martin wrote:
> >> > On Jul 27, 12:42 pm, Peter Otten <(E-Mail Removed)> wrote:
> >> >> Martin wrote:
> >> >> > I am new to python and I was wondering if there was a way to speed
> >> >> > up the way I index 2D arrays when I need to check two arrays
> >> >> > simultaneously? My current implementations is (using numpy)
> >> >> > something like the following...

>
> >> >> > for i in range(numrows):
> >> >> > for j in range(numcols):

>
> >> >> > if array_1[i, j] == some_value or array_2[i, j] >=
> >> >> > array_1[i,
> >> >> > j] * some_other_value
> >> >> > array_1[i, j] = some_new_value

>
> >> >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
> >> >> some_other_value)] = some_new_value

>
> >> >> maybe?

>
> >> > So I tried...

>
> >> > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

>
> >> > which led to

>
> >> > ValueError: The truth value of an array with more than one element is
> >> > ambiguous. Use a.any() or a.all()

>
> >> > so not sure that works?

>
> >> Copy and paste -- or replace "or" with "|".

>
> > apologies - I mistook that for a type for "or"

>
> > I now get the following error...

>
> > ValueError: shape mismatch: objects cannot be broadcast to a single
> > shape

>
> It seems array_1 and array_2 have a -- dada! -- different shape.
> Assuming array_1 is the smaller one:
>
> numrows, numcols = array_1.shape
> array_1[(array_1 == some_value) | (array_2[:numrows,:numcols] >= array_1 *
> some_other_value)] = some_new_value
>
> There may be other options, but I'm not a numpy expert.
>
> Peter


My mistake - the incorrect size in the arrays was my error.

The statement works now, but it doesn't give the same results as my
original logic, strangely!?

in my logic:

data = np.zeros((numrows, numcols), dtype = np.uint8, order ='C')

for i in range(numrows):
for j in range(numcols):
if band3[i,j] == 255 or band3[i,j] >= band6[i,j] * factor:
data[i,j] = 0
else:
data[i,j] = 1

to do the same with what you suggested...

data = np.ones((numrows, numcols), dtype = np.uint8, order ='C')
data[(band3 == 255) | (band6 >= band3 * factor)] = 0

Thanks
 
Reply With Quote
 
Peter Otten
Guest
Posts: n/a
 
      07-27-2009
Martin wrote:

> The statement works now, but it doesn't give the same results as my
> original logic, strangely!?
>
> in my logic:
>
> data = np.zeros((numrows, numcols), dtype = np.uint8, order ='C')
>
> for i in range(numrows):
> for j in range(numcols):
> if band3[i,j] == 255 or band3[i,j] >= band6[i,j] * factor:
> data[i,j] = 0
> else:
> data[i,j] = 1
>
> to do the same with what you suggested...
>
> data = np.ones((numrows, numcols), dtype = np.uint8, order ='C')
> data[(band3 == 255) | (band6 >= band3 * factor)] = 0


Did you swap band3 and band6? If that's the case, it is an error you should
be able to find yourself.

Please be a bit more careful.

Also, it is good practice to write a few test cases where you have
precalculated the result. How would you otherwise know which version is
correct? Writing a third one to break the tie?

Peter


 
Reply With Quote
 
Martin
Guest
Posts: n/a
 
      07-27-2009
On Jul 27, 4:12 pm, Peter Otten <(E-Mail Removed)> wrote:
> Martin wrote:
> > The statement works now, but it doesn't give the same results as my
> > original logic, strangely!?

>
> > in my logic:

>
> > data = np.zeros((numrows, numcols), dtype = np.uint8, order ='C')

>
> > for i in range(numrows):
> > for j in range(numcols):
> > if band3[i,j] == 255 or band3[i,j] >= band6[i,j] * factor:
> > data[i,j] = 0
> > else:
> > data[i,j] = 1

>
> > to do the same with what you suggested...

>
> > data = np.ones((numrows, numcols), dtype = np.uint8, order ='C')
> > data[(band3 == 255) | (band6 >= band3 * factor)] = 0

>
> Did you swap band3 and band6? If that's the case, it is an error you should
> be able to find yourself.
>
> Please be a bit more careful.
>
> Also, it is good practice to write a few test cases where you have
> precalculated the result. How would you otherwise know which version is
> correct? Writing a third one to break the tie?
>
> Peter


apologies... it was the way I typed it up, I wasn't copying and
pasting from my text editor!
Thanks Peter.
 
Reply With Quote
 
Robert Kern
Guest
Posts: n/a
 
      07-27-2009
On 2009-07-27 06:24, Martin wrote:
> Hi,
>
> I am new to python and I was wondering if there was a way to speed up
> the way I index 2D arrays when I need to check two arrays
> simultaneously? My current implementations is (using numpy) something
> like the following...
>
> for i in range(numrows):
> for j in range(numcols):
>
> if array_1[i, j] == some_value or array_2[i, j]>= array_1[i,
> j] * some_other_value
> array_1[i, j] = some_new_value


Peter has given you good answers, but you will probably want to ask future numpy
questions on the numpy mailing list.

http://www.scipy.org/Mailing_Lists

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
looping in array vs looping in a dic giuseppe.amatulli@gmail.com Python 5 09-20-2012 11:58 PM
VOIP over VPN over TCP over WAP over 3G Theo Markettos UK VOIP 2 02-14-2008 03:27 PM
Berkely Db. How to iterate over large number of keys "quickly" lazy Python 5 08-03-2007 02:03 AM
Looping over my WebControls Jim Bancroft ASP .Net 7 07-05-2005 10:20 PM
Looping over a DataGrid after postback; questions Jim Bancroft ASP .Net 2 12-31-2004 04:39 PM



Advertisments