Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   Re: C API: array of floats/ints from python to C and back (http://www.velocityreviews.com/forums/t650297-re-c-api-array-of-floats-ints-from-python-to-c-and-back.html)

Daniel Fetchinson 12-27-2008 10:27 PM

Re: C API: array of floats/ints from python to C and back
 
>> I'm trying to write an extension module in C which contains a single
>> function with the following prototype:
>> void func( int N, int * arg1, int * arg2, int * ret );
>> Here arg1 and arg2 are length N arrays, and the function computes ret
>> which is also an N length array. From python I'd like to call this
>> function as
>> ret = func( [ 1, 2, 3 ], [ 2, 3, 4] )

>
> This requirement pretty much dictates the slow answer you have.


I'm not sure I understand what you mean.

> > Does this mean that I can only pass the arrays from python to C as
> > generic python objects and in a later operation I need to get the
> > elements from this generic python object, construct a C array and pass
> > that to the C function? ... What's the simplest way of doing this?

>
> Either use ctypes, look into array.array, or look into numpy.array.


I have considered using ctypes but for my needs using the C API
directly seems more reasonable. array.array and numpy.array doesn't
fit my needs since I need to do long and complicated operations on the
two (pretty large) integer arrays that would be too slow using
array.array and numpy.array (I've verified this claim by benchmarking
a numpy.array based solution).

> I'd just use numpy, myself:
> import numpy
> total = numpy.array([1, 2, 3]) + numpy.array([2, 3, 4])


What makes you think I want to add two arrays?

Cheers,
Daniel


--
Psss, psss, put it down! - http://www.cafepress.com/putitdown

Robert Kern 12-28-2008 12:37 AM

Re: C API: array of floats/ints from python to C and back
 
Daniel Fetchinson wrote:

> I agree that array.array is more efficient than a list but the input
> for my function will come from PIL and PIL returns a list. So I have a
> list to begin with which will be passed to the C function.


With recent versions of PIL, numpy can create an array from an Image very
quickly, possibly without any copying of memory.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco


Aaron Brady 12-28-2008 12:40 AM

Re: C API: array of floats/ints from python to C and back
 
On Dec 27, 6:06*pm, Scott David Daniels <Scott.Dani...@Acm.Org> wrote:
> Daniel Fetchinson wrote:
> >.... I have a list to begin with which will be passed to the C function.

>
> *> I assume converting the list to an array.array and passing that to the C
>
> > function doesn't make any difference in terms of speed since the
> > operation itself will be done in the C function anyway.

>
> Right, but why bother to do the conversion in C where you'll have to
> fiddle with refcounts and error propogation? *convert in python, and
> go to the underlying data in C.


I think you could use ctypes.c_int* 20, which is an awesome data
type-- a 20-element array of integers. You could also use this, which
I just discovered, and is actually moderately disturbing.

#WARNING unsafe code
>>> from ctypes import *
>>> i= c_int(42)
>>> pi= pointer(i)
>>> pi[0]

42
>>> pi[1]

0
>>> pi[2]

0
>>> pi[1]= 20
>>> pi

<__main__.LP_c_long object at 0x00A97D00>
>>> pi[1]

20

What memory is it overwriting?

Daniel Fetchinson 12-28-2008 12:44 AM

Re: C API: array of floats/ints from python to C and back
 
>> I agree that array.array is more efficient than a list but the input
>> for my function will come from PIL and PIL returns a list. So I have a
>> list to begin with which will be passed to the C function.

>
> With recent versions of PIL, numpy can create an array from an Image very
> quickly, possibly without any copying of memory.


What exactly do you mean? (1) PIL creates a list which can be easily
converted by numpy to a numpy.array or (2) with the help of numpy one
can create a numpy.array from an image directly?

Since I will have to pass the list or numy.array to C anyway I don't
see any advantage to (1) although (2) can be useful.

Daniel

--
Psss, psss, put it down! - http://www.cafepress.com/putitdown

Daniel Fetchinson 12-28-2008 12:54 AM

Re: C API: array of floats/ints from python to C and back
 
On 12/27/08, Robert Kern <robert.kern@gmail.com> wrote:
> Daniel Fetchinson wrote:
>
>> I agree that array.array is more efficient than a list but the input
>> for my function will come from PIL and PIL returns a list. So I have a
>> list to begin with which will be passed to the C function.

>
> With recent versions of PIL, numpy can create an array from an Image very
> quickly, possibly without any copying of memory.
>
> --
> Robert Kern
>
> "I have come to believe that the whole world is an enigma, a harmless enigma
> that is made terrible by our own mad attempt to interpret it as though it
> had
> an underlying truth."
> -- Umberto Eco
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>


Hi Robert, now that we are at it let me ask something related. Now I
managed to get my lists to C, operate on them in C and convert the
result back to a python tuple. Since this is my first C program that
uses the python C API I'm sure there are refcounting issues that I'm
not considering, maybe also memory management too.

This is the function I have, the corresponding python function will
take two equal length lists of integers and the C function will
compute their sum and return the result as a python tuple.


static PyObject *func( PyObject * self, PyObject * args )
{
int j, N;
int * src1, * src2;
PyObject *list1, *list2;

list1 = PyTuple_GetItem( args, 0 );
N = PyList_Size( list1 );
src1 = ( int * ) malloc( N * sizeof( int ) );
for( j = 0; j < N; j++ )
{
src1[j] = (int)PyInt_AsLong( PyList_GetItem( list1, j ) );
}

list2 = PyTuple_GetItem( args, 1 );
N = PyList_Size( list2 );
src2 = ( int * ) malloc( N * sizeof( int ) );
for( j = 0; j < N; j++ )
{
src2[j] = (int)PyInt_AsLong( PyList_GetItem( list2, j ) );
}

PyObject * tuple;
tuple = PyTuple_New( N );
for( j = 0; j < N; j++ )
{
PyTuple_SetItem( tuple, j, PyInt_FromLong( (long)( src1[j] +
src2[j] ) ) );
}

free( src1 );
free( src2 );

return tuple;
}

Do I have to free the memory occupied by the python objects list1 and
list2? Do I have to do any refcounting for list1, list2, tuple?

Any comment on the above code will be very appreciated! If I'm pushed
in the right direction I'm a fast learner but the beginning steps are
always hard :)

Cheers,
Daniel



--
Psss, psss, put it down! - http://www.cafepress.com/putitdown

Gabriel Genellina 12-28-2008 01:59 AM

Re: C API: array of floats/ints from python to C and back
 
En Sat, 27 Dec 2008 22:54:52 -0200, Daniel Fetchinson
<fetchinson@googlemail.com> escribió:

> This is the function I have, the corresponding python function will
> take two equal length lists of integers and the C function will
> compute their sum and return the result as a python tuple.
>
>
> static PyObject *func( PyObject * self, PyObject * args )
> {
> int j, N;
> int * src1, * src2;
> PyObject *list1, *list2;
>
> list1 = PyTuple_GetItem( args, 0 );
> N = PyList_Size( list1 );
> src1 = ( int * ) malloc( N * sizeof( int ) );
> for( j = 0; j < N; j++ )
> {
> src1[j] = (int)PyInt_AsLong( PyList_GetItem( list1, j ) );
> }
>
> list2 = PyTuple_GetItem( args, 1 );
> N = PyList_Size( list2 );
> src2 = ( int * ) malloc( N * sizeof( int ) );
> for( j = 0; j < N; j++ )
> {
> src2[j] = (int)PyInt_AsLong( PyList_GetItem( list2, j ) );
> }
>
> PyObject * tuple;
> tuple = PyTuple_New( N );
> for( j = 0; j < N; j++ )
> {
> PyTuple_SetItem( tuple, j, PyInt_FromLong( (long)( src1[j] +
> src2[j] ) ) );
> }
>
> free( src1 );
> free( src2 );
>
> return tuple;
> }


As others already said, using a Numpy array or an array.array object would
be more efficient (and even easier - the C code gets a pointer to an array
of integers, as usual).

> Do I have to free the memory occupied by the python objects list1 and
> list2?


No. Usually you don't do that for any Python object - just
increment/decrement its reference count (using Py_INCREF/Py_DECREF).

> Do I have to do any refcounting for list1, list2, tuple?


In this case list1 and list2 come from PyTuple_GetItem; the docs say it
returns a "borrowed reference" (that is, the function doesn't increment
the refcount itself). So you don't have to decrement it yourself (and it
isn't necesary to increment it in the first place, because the "args"
tuple holds a reference, so the object can't disappear until the function
exits, at least)

> Any comment on the above code will be very appreciated! If I'm pushed
> in the right direction I'm a fast learner but the beginning steps are
> always hard :)


You MUST check EVERY function call for errors!
And check the argument's type (how do you know it is a list?). Once you
are sure the first parameter *is* a list, you may use the "macro" version
of several functions, like PyList_GET_SIZE and PyList_GET_ITEM.
You should check that both lists have the same length too.
And you should check that elements are integers, or convertible to
integers (in case of error, PyInt_AsLong returns -1 and PyErr_Occurred()
is true)
To fill the resulting tuple, use PyTuple_SET_ITEM instead. BTW, why return
a tuple and not a list?

--
Gabriel Genellina


Robert Kern 12-28-2008 02:39 AM

Re: C API: array of floats/ints from python to C and back
 
Daniel Fetchinson wrote:
>>> I agree that array.array is more efficient than a list but the input
>>> for my function will come from PIL and PIL returns a list. So I have a
>>> list to begin with which will be passed to the C function.

>> With recent versions of PIL, numpy can create an array from an Image very
>> quickly, possibly without any copying of memory.

>
> What exactly do you mean? (1) PIL creates a list which can be easily
> converted by numpy to a numpy.array or (2) with the help of numpy one
> can create a numpy.array from an image directly?


(2) a = numpy.asarray(img)

> Since I will have to pass the list or numy.array to C anyway I don't
> see any advantage to (1) although (2) can be useful.


(1) is still somewhat useful, if you're willing to bear the cost of the numpy
dependency. The conversion code from any Python sequence to a numpy array of the
desired type and dimensionality is a 2-liner (function call and error check).
The memory is managed by numpy, so all you have to do is manage the refcount of
the array object like any other Python object.

Okay, 5-liner given C's verbosity:

intx = PyArray_FROM_OTF(pyintx, PyArray_INT, NPY_IN_ARRAY);
if (!intx) {
PyErr_SetString(PyExc_ValueError, "intx must be an array of ints");
goto fail;
}

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco


Daniel Fetchinson 12-28-2008 02:40 AM

Re: C API: array of floats/ints from python to C and back
 
>> This is the function I have, the corresponding python function will
>> take two equal length lists of integers and the C function will
>> compute their sum and return the result as a python tuple.
>>
>>
>> static PyObject *func( PyObject * self, PyObject * args )
>> {
>> int j, N;
>> int * src1, * src2;
>> PyObject *list1, *list2;
>>
>> list1 = PyTuple_GetItem( args, 0 );
>> N = PyList_Size( list1 );
>> src1 = ( int * ) malloc( N * sizeof( int ) );
>> for( j = 0; j < N; j++ )
>> {
>> src1[j] = (int)PyInt_AsLong( PyList_GetItem( list1, j ) );
>> }
>>
>> list2 = PyTuple_GetItem( args, 1 );
>> N = PyList_Size( list2 );
>> src2 = ( int * ) malloc( N * sizeof( int ) );
>> for( j = 0; j < N; j++ )
>> {
>> src2[j] = (int)PyInt_AsLong( PyList_GetItem( list2, j ) );
>> }
>>
>> PyObject * tuple;
>> tuple = PyTuple_New( N );
>> for( j = 0; j < N; j++ )
>> {
>> PyTuple_SetItem( tuple, j, PyInt_FromLong( (long)( src1[j] +
>> src2[j] ) ) );
>> }
>>
>> free( src1 );
>> free( src2 );
>>
>> return tuple;
>> }

>
> As others already said, using a Numpy array or an array.array object would
> be more efficient (and even easier - the C code gets a pointer to an array
> of integers, as usual).


Thanks, I didn't know that an array.array can be passed to C code as a
C pointer to an array. So I'll use an array.array because this is
really convenient.

>> Do I have to free the memory occupied by the python objects list1 and
>> list2?

>
> No. Usually you don't do that for any Python object - just
> increment/decrement its reference count (using Py_INCREF/Py_DECREF).


Great, one headache less :)

>> Do I have to do any refcounting for list1, list2, tuple?

>
> In this case list1 and list2 come from PyTuple_GetItem; the docs say it
> returns a "borrowed reference" (that is, the function doesn't increment
> the refcount itself). So you don't have to decrement it yourself (and it
> isn't necesary to increment it in the first place, because the "args"
> tuple holds a reference, so the object can't disappear until the function
> exits, at least)
>
>> Any comment on the above code will be very appreciated! If I'm pushed
>> in the right direction I'm a fast learner but the beginning steps are
>> always hard :)

>
> You MUST check EVERY function call for errors!


Yes, I know :)

> And check the argument's type (how do you know it is a list?). Once you
> are sure the first parameter *is* a list, you may use the "macro" version
> of several functions, like PyList_GET_SIZE and PyList_GET_ITEM.


The macro versions are preferable because they are faster?

> You should check that both lists have the same length too.
> And you should check that elements are integers, or convertible to
> integers (in case of error, PyInt_AsLong returns -1 and PyErr_Occurred()
> is true)
> To fill the resulting tuple, use PyTuple_SET_ITEM instead. BTW, why return
> a tuple and not a list?


No particular reason, other than the fact that I won't need to modify
these lists/tuples from python so whenever something will not change,
I use a tuple because it's immutable. Or this is not a very good
practice? There is no difference between lists and tuples in terms of
speed I suppose (getitem, setitem, etc).

Thanks a lot,
Daniel


--
Psss, psss, put it down! - http://www.cafepress.com/putitdown

Daniel Fetchinson 12-28-2008 02:44 AM

Re: C API: array of floats/ints from python to C and back
 
>>>> I agree that array.array is more efficient than a list but the input
>>>> for my function will come from PIL and PIL returns a list. So I have a
>>>> list to begin with which will be passed to the C function.
>>> With recent versions of PIL, numpy can create an array from an Image very
>>> quickly, possibly without any copying of memory.

>>
>> What exactly do you mean? (1) PIL creates a list which can be easily
>> converted by numpy to a numpy.array or (2) with the help of numpy one
>> can create a numpy.array from an image directly?

>
> (2) a = numpy.asarray(img)


Thanks, I didn't know about this, maybe it will be useful.

>> Since I will have to pass the list or numy.array to C anyway I don't
>> see any advantage to (1) although (2) can be useful.

>
> (1) is still somewhat useful, if you're willing to bear the cost of the
> numpy
> dependency. The conversion code from any Python sequence to a numpy array of
> the
> desired type and dimensionality is a 2-liner (function call and error
> check).
> The memory is managed by numpy, so all you have to do is manage the refcount
> of
> the array object like any other Python object.
>
> Okay, 5-liner given C's verbosity:
>
> intx = PyArray_FROM_OTF(pyintx, PyArray_INT, NPY_IN_ARRAY);
> if (!intx) {
> PyErr_SetString(PyExc_ValueError, "intx must be an array of ints");
> goto fail;
> }


Yes, the dependency on numpy is my main concern. If it will help with
my problem I don't mind the dependency actually, so I'll do more
detailed benchmarks first.

Thank you,
Daniel



--
Psss, psss, put it down! - http://www.cafepress.com/putitdown

Gabriel Genellina 12-28-2008 03:38 AM

Re: C API: array of floats/ints from python to C and back
 
En Sun, 28 Dec 2008 00:40:52 -0200, Daniel Fetchinson
<fetchinson@googlemail.com> escribió:

>> You MUST check EVERY function call for errors!

>
> Yes, I know :)
>


Believe me, if you don't, there is a risk of crashing the program. And
they're a lot harder to find and fix.

>> And check the argument's type (how do you know it is a list?). Once you
>> are sure the first parameter *is* a list, you may use the "macro"
>> version
>> of several functions, like PyList_GET_SIZE and PyList_GET_ITEM.

>
> The macro versions are preferable because they are faster?


Yes, because they don't do any additional checking. PyList_GET_ITEM just
retrieves the item; PyList_GetItem checks whether it is called on a list
object, then checks if index is out of bounds, and only then goes to
retrieve the item.

>> You should check that both lists have the same length too.
>> And you should check that elements are integers, or convertible to
>> integers (in case of error, PyInt_AsLong returns -1 and PyErr_Occurred()
>> is true)
>> To fill the resulting tuple, use PyTuple_SET_ITEM instead. BTW, why
>> return
>> a tuple and not a list?

>
> No particular reason, other than the fact that I won't need to modify
> these lists/tuples from python so whenever something will not change,
> I use a tuple because it's immutable. Or this is not a very good
> practice? There is no difference between lists and tuples in terms of
> speed I suppose (getitem, setitem, etc).


Usually lists represent an ordered collection of similar items, where
position is not relevant (the third item is treated more or less the same
as the tenth); by example, a list of attendants to certain event. It makes
sense to process the whole list doing the same thing to each element, and
it makes sense to ask "is this item in the list?."

Tuples represent an ordered collection of dissimilar items, where position
is relevant (because it identifies each item); by example, a row from a
database table (name, address, age, phone number), or a point (x,y,z) in
space. It isn't common to process the whole tuple doing the same thing for
each element, and usually it doesn't make sense to look for certain item
in the tuple. But it does make sense to refer to individual items like
t[2], or t.age (namedtuple, 2.6 and up)

From this point of view, lists and tuples are conceptually different -
tuples aren't "frozen" lists. But this is just common usage, or maybe
historical intent; of course you are free to use whatever structure you
feel adequate.

Once the list/tuple is created and filled, there is no speed difference
accessing the individual items. Creating an empty list that grows one
element at a time is slow for large lists (the memory block has to be
re-allocated and copied over evry time it gets full) but this doesn't
happen if you provide the final size when creating the list.

--
Gabriel Genellina



All times are GMT. The time now is 12:41 AM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.