Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Auto free Buffer objects in C API?

Reply
Thread Tools

Auto free Buffer objects in C API?

 
 
Stuart D. Gathman
Guest
Posts: n/a
 
      07-03-2003
I have a Python C module (dspam.c - http://bmsi.com/python/milter.html).
It needs to return some large buffers allocated by the library I am
wrapping. The buffers must be freed by the caller when finished. At
present, I do the following to copy, then free the buffer:

PyObject *
toBuffer(char *buf;int len) {
PyObject *sig = PyBuffer_New(len);
if (sig) {
void *data;
int dlen;
if (!PyObject_AsWriteBuffer(sig,&data,&dlen))
memcpy(data,buf,dlen);
else {
Py_DECREF(sig);
sig = 0;
}
}
free(buf);
return sig;
}

Now, I would like to use PyBuffer_FromMemory(buf,len), but then I have no
way of knowing when to free(buf). Or do I? Am I missing something? Is
there a way for me to know when the Buffer object is being deleted so
that I can free the underlying C data?
 
Reply With Quote
 
 
 
 
David M. Cooke
Guest
Posts: n/a
 
      07-04-2003
At some point, "Stuart D. Gathman" <(E-Mail Removed)> wrote:

> I have a Python C module (dspam.c - http://bmsi.com/python/milter.html).
> It needs to return some large buffers allocated by the library I am
> wrapping. The buffers must be freed by the caller when finished. At
> present, I do the following to copy, then free the buffer:
>
> PyObject *
> toBuffer(char *buf;int len) {
> PyObject *sig = PyBuffer_New(len);
> if (sig) {
> void *data;
> int dlen;
> if (!PyObject_AsWriteBuffer(sig,&data,&dlen))
> memcpy(data,buf,dlen);
> else {
> Py_DECREF(sig);
> sig = 0;
> }
> }
> free(buf);
> return sig;
> }
>
> Now, I would like to use PyBuffer_FromMemory(buf,len), but then I have no
> way of knowing when to free(buf). Or do I? Am I missing something? Is
> there a way for me to know when the Buffer object is being deleted so
> that I can free the underlying C data?


Staring at the definition of PyBufferObject in bufferobject.c, I see
it has a member 'b_base' of type PyObject *, which is DECREF'd when
the buffer is destroyed (by the buffer dealloc function). Looks to me
like you could do something using a CObject with a passed destructor:

/**********/
/* WARNING: UNTESTED CODE */

/* the contents of PyBufferObject are supposed to be private, so you
need to include this */
typedef struct {
PyObject_HEAD
PyObject *b_base;
void *b_ptr;
int b_size;
int b_readonly;
long b_hash;
} PyBufferObject;

PyObject *
toBuffer(char *buf, int len) {
PyObject *sig = PyBuffer_FromMemory(buf, len);
if (sig) {
/* pass 'free' as the destructor for the passed memory */
(PyBufferObject *)sig->b_base = PyCObject_FromVoidPtr(buf, free);
}
return sig;
}
/**********/

Now, when the buffer is deallocated, the CObject at b_base will be
DECREF'd, which will free your memory.

This is a bit ugly, as it depends on internal details. Another,
cleaner, method is to define an object which obeys the buffer
protocol, and handles destruction like you want. (This is annoying, as
you'll end up copying all the code in bufferobject.c).

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)physics(dot)mcmaster(dot)ca
 
Reply With Quote
 
 
 
 
Stuart D. Gathman
Guest
Posts: n/a
 
      07-07-2003
On Thu, 03 Jul 2003 21:44:27 -0400, David M. Cooke wrote:

> This is a bit ugly, as it depends on internal details. Another, cleaner,
> method is to define an object which obeys the buffer protocol, and
> handles destruction like you want. (This is annoying, as you'll end up
> copying all the code in bufferobject.c).


So annoying, in fact, that I will continue to make extra copies of the
buffer until such time as:

a) Python gets a PyBuffer_FromCObject(PyObject *,int len) method.

b) Profiling reveals the extra copies to be a major bottle neck. The
extra memory allocation is not a problem because the library spends most
of its time moving the data between two buffers while processing it.

BTW, the application is wrapping DSpam for Python. You can download the
dspam module from:

http://bmsi.com/python/milter.html
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Auto Shipping Auto Shipping Scheduling:car moving auto transport linkswanted ASP .Net 1 11-22-2013 07:02 AM
Free Online Video for Nikon D200 on Auto Exposure and Auto Focus Lock elitevideo1@gmail.com Digital Photography 0 04-03-2006 09:17 PM
How to know the buffer size and increase buffer size in c++ Raja C++ 12 06-21-2004 06:21 PM



Advertisments