Velocity Reviews > quickly looping over a 2D array?

quickly looping over a 2D array?

Martin
Guest
Posts: n/a

 07-27-2009
Hi,

I am new to python and I was wondering if there was a way to speed up
the way I index 2D arrays when I need to check two arrays
simultaneously? My current implementations is (using numpy) something
like the following...

for i in range(numrows):
for j in range(numcols):

if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
j] * some_other_value
array_1[i, j] = some_new_value

Many Thanks,

Martin

Peter Otten
Guest
Posts: n/a

 07-27-2009
Martin wrote:

> I am new to python and I was wondering if there was a way to speed up
> the way I index 2D arrays when I need to check two arrays
> simultaneously? My current implementations is (using numpy) something
> like the following...
>
> for i in range(numrows):
> for j in range(numcols):
>
> if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
> j] * some_other_value
> array_1[i, j] = some_new_value

array_1[(array_1 == some_value) | (array_2 >= array_1 * some_other_value)] =
some_new_value

maybe?

Martin
Guest
Posts: n/a

 07-27-2009
On Jul 27, 12:42 pm, Peter Otten <(E-Mail Removed)> wrote:
> Martin wrote:
> > I am new to python and I was wondering if there was a way to speed up
> > the way I index 2D arrays when I need to check two arrays
> > simultaneously? My current implementations is (using numpy) something
> > like the following...

>
> > for i in range(numrows):
> > for j in range(numcols):

>
> > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
> > j] * some_other_value
> > array_1[i, j] = some_new_value

>
> array_1[(array_1 == some_value) | (array_2 >= array_1 * some_other_value)] =
> some_new_value
>
> maybe?

So I tried...

band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

which led to

ValueError: The truth value of an array with more than one element is
ambiguous. Use a.any() or a.all()

so not sure that works?

Peter Otten
Guest
Posts: n/a

 07-27-2009
Martin wrote:

> On Jul 27, 12:42 pm, Peter Otten <(E-Mail Removed)> wrote:
>> Martin wrote:
>> > I am new to python and I was wondering if there was a way to speed up
>> > the way I index 2D arrays when I need to check two arrays
>> > simultaneously? My current implementations is (using numpy) something
>> > like the following...

>>
>> > for i in range(numrows):
>> > for j in range(numcols):

>>
>> > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
>> > j] * some_other_value
>> > array_1[i, j] = some_new_value

>>
>> array_1[(array_1 == some_value) | (array_2 >= array_1 *
>> some_other_value)] = some_new_value
>>
>> maybe?

>
> So I tried...
>
> band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0
>
> which led to
>
> ValueError: The truth value of an array with more than one element is
> ambiguous. Use a.any() or a.all()
>
> so not sure that works?

Copy and paste -- or replace "or" with "|".

Martin
Guest
Posts: n/a

 07-27-2009
On Jul 27, 1:46 pm, Peter Otten <(E-Mail Removed)> wrote:
> Martin wrote:
> > On Jul 27, 12:42 pm, Peter Otten <(E-Mail Removed)> wrote:
> >> Martin wrote:
> >> > I am new to python and I was wondering if there was a way to speed up
> >> > the way I index 2D arrays when I need to check two arrays
> >> > simultaneously? My current implementations is (using numpy) something
> >> > like the following...

>
> >> > for i in range(numrows):
> >> > for j in range(numcols):

>
> >> > if array_1[i, j] == some_value or array_2[i, j] >= array_1[i,
> >> > j] * some_other_value
> >> > array_1[i, j] = some_new_value

>
> >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
> >> some_other_value)] = some_new_value

>
> >> maybe?

>
> > So I tried...

>
> > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

>
> > which led to

>
> > ValueError: The truth value of an array with more than one element is
> > ambiguous. Use a.any() or a.all()

>
> > so not sure that works?

>
> Copy and paste -- or replace "or" with "|".

apologies - I mistook that for a type for "or"

I now get the following error...

ValueError: shape mismatch: objects cannot be broadcast to a single
shape

Peter Otten
Guest
Posts: n/a

 07-27-2009
Martin wrote:

> On Jul 27, 1:46 pm, Peter Otten <(E-Mail Removed)> wrote:
>> Martin wrote:
>> > On Jul 27, 12:42 pm, Peter Otten <(E-Mail Removed)> wrote:
>> >> Martin wrote:
>> >> > I am new to python and I was wondering if there was a way to speed
>> >> > up the way I index 2D arrays when I need to check two arrays
>> >> > simultaneously? My current implementations is (using numpy)
>> >> > something like the following...

>>
>> >> > for i in range(numrows):
>> >> > for j in range(numcols):

>>
>> >> > if array_1[i, j] == some_value or array_2[i, j] >=
>> >> > array_1[i,
>> >> > j] * some_other_value
>> >> > array_1[i, j] = some_new_value

>>
>> >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
>> >> some_other_value)] = some_new_value

>>
>> >> maybe?

>>
>> > So I tried...

>>
>> > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

>>
>> > which led to

>>
>> > ValueError: The truth value of an array with more than one element is
>> > ambiguous. Use a.any() or a.all()

>>
>> > so not sure that works?

>>
>> Copy and paste -- or replace "or" with "|".

>
> apologies - I mistook that for a type for "or"
>
> I now get the following error...
>
> ValueError: shape mismatch: objects cannot be broadcast to a single
> shape

It seems array_1 and array_2 have a -- dada! -- different shape.
Assuming array_1 is the smaller one:

numrows, numcols = array_1.shape
array_1[(array_1 == some_value) | (array_2[:numrows,:numcols] >= array_1 *
some_other_value)] = some_new_value

There may be other options, but I'm not a numpy expert.

Peter

Martin
Guest
Posts: n/a

 07-27-2009
On Jul 27, 2:17 pm, Peter Otten <(E-Mail Removed)> wrote:
> Martin wrote:
> > On Jul 27, 1:46 pm, Peter Otten <(E-Mail Removed)> wrote:
> >> Martin wrote:
> >> > On Jul 27, 12:42 pm, Peter Otten <(E-Mail Removed)> wrote:
> >> >> Martin wrote:
> >> >> > I am new to python and I was wondering if there was a way to speed
> >> >> > up the way I index 2D arrays when I need to check two arrays
> >> >> > simultaneously? My current implementations is (using numpy)
> >> >> > something like the following...

>
> >> >> > for i in range(numrows):
> >> >> > for j in range(numcols):

>
> >> >> > if array_1[i, j] == some_value or array_2[i, j] >=
> >> >> > array_1[i,
> >> >> > j] * some_other_value
> >> >> > array_1[i, j] = some_new_value

>
> >> >> array_1[(array_1 == some_value) | (array_2 >= array_1 *
> >> >> some_other_value)] = some_new_value

>
> >> >> maybe?

>
> >> > So I tried...

>
> >> > band_1[(array_1 == 255) or (array_2 >= array_1 * factor)] = 0

>
> >> > which led to

>
> >> > ValueError: The truth value of an array with more than one element is
> >> > ambiguous. Use a.any() or a.all()

>
> >> > so not sure that works?

>
> >> Copy and paste -- or replace "or" with "|".

>
> > apologies - I mistook that for a type for "or"

>
> > I now get the following error...

>
> > ValueError: shape mismatch: objects cannot be broadcast to a single
> > shape

>
> It seems array_1 and array_2 have a -- dada! -- different shape.
> Assuming array_1 is the smaller one:
>
> numrows, numcols = array_1.shape
> array_1[(array_1 == some_value) | (array_2[:numrows,:numcols] >= array_1 *
> some_other_value)] = some_new_value
>
> There may be other options, but I'm not a numpy expert.
>
> Peter

My mistake - the incorrect size in the arrays was my error.

The statement works now, but it doesn't give the same results as my
original logic, strangely!?

in my logic:

data = np.zeros((numrows, numcols), dtype = np.uint8, order ='C')

for i in range(numrows):
for j in range(numcols):
if band3[i,j] == 255 or band3[i,j] >= band6[i,j] * factor:
data[i,j] = 0
else:
data[i,j] = 1

to do the same with what you suggested...

data = np.ones((numrows, numcols), dtype = np.uint8, order ='C')
data[(band3 == 255) | (band6 >= band3 * factor)] = 0

Thanks

Peter Otten
Guest
Posts: n/a

 07-27-2009
Martin wrote:

> The statement works now, but it doesn't give the same results as my
> original logic, strangely!?
>
> in my logic:
>
> data = np.zeros((numrows, numcols), dtype = np.uint8, order ='C')
>
> for i in range(numrows):
> for j in range(numcols):
> if band3[i,j] == 255 or band3[i,j] >= band6[i,j] * factor:
> data[i,j] = 0
> else:
> data[i,j] = 1
>
> to do the same with what you suggested...
>
> data = np.ones((numrows, numcols), dtype = np.uint8, order ='C')
> data[(band3 == 255) | (band6 >= band3 * factor)] = 0

Did you swap band3 and band6? If that's the case, it is an error you should
be able to find yourself.

Please be a bit more careful.

Also, it is good practice to write a few test cases where you have
precalculated the result. How would you otherwise know which version is
correct? Writing a third one to break the tie?

Peter

Martin
Guest
Posts: n/a

 07-27-2009
On Jul 27, 4:12 pm, Peter Otten <(E-Mail Removed)> wrote:
> Martin wrote:
> > The statement works now, but it doesn't give the same results as my
> > original logic, strangely!?

>
> > in my logic:

>
> > data = np.zeros((numrows, numcols), dtype = np.uint8, order ='C')

>
> > for i in range(numrows):
> > for j in range(numcols):
> > if band3[i,j] == 255 or band3[i,j] >= band6[i,j] * factor:
> > data[i,j] = 0
> > else:
> > data[i,j] = 1

>
> > to do the same with what you suggested...

>
> > data = np.ones((numrows, numcols), dtype = np.uint8, order ='C')
> > data[(band3 == 255) | (band6 >= band3 * factor)] = 0

>
> Did you swap band3 and band6? If that's the case, it is an error you should
> be able to find yourself.
>
> Please be a bit more careful.
>
> Also, it is good practice to write a few test cases where you have
> precalculated the result. How would you otherwise know which version is
> correct? Writing a third one to break the tie?
>
> Peter

apologies... it was the way I typed it up, I wasn't copying and
pasting from my text editor!
Thanks Peter.

Robert Kern
Guest
Posts: n/a

 07-27-2009
On 2009-07-27 06:24, Martin wrote:
> Hi,
>
> I am new to python and I was wondering if there was a way to speed up
> the way I index 2D arrays when I need to check two arrays
> simultaneously? My current implementations is (using numpy) something
> like the following...
>
> for i in range(numrows):
> for j in range(numcols):
>
> if array_1[i, j] == some_value or array_2[i, j]>= array_1[i,
> j] * some_other_value
> array_1[i, j] = some_new_value

Peter has given you good answers, but you will probably want to ask future numpy
questions on the numpy mailing list.

http://www.scipy.org/Mailing_Lists

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco