Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Python C API String Memory Consumption

Reply
Thread Tools

Python C API String Memory Consumption

 
 
k3xji
Guest
Posts: n/a
 
      04-07-2009
When I run the following function, I seem to have a mem leak, a 20 mb
of memory
is allocated and is not freed. Here is the code I run:

>>> import esauth
>>> for i in range(1000000):


.... ss = esauth.penc('sumer')
....
>>> for i in range(1000000):


.... ss = esauth.penc('sumer')
....

And here is the penc() function.

static PyObject *
penc(PyObject *self, PyObject *args)
{
unsigned char *s= NULL;
unsigned char *buf = NULL;
PyObject * result = NULL;
unsigned int v,len,i = 0;

if (!PyArg_ParseTuple(args, "s#", &s, &len))
return NULL;

buf = strdup(s);
if (!buf) {
PyErr_SetString(PyExc_MemoryError,
"Out of memory: strdup failed");
return NULL;
}

/*string manipulation*/

result = PyString_FromString(buf);
free(buf);
return result;

}

Am I doing something wrong?

Thanks,
 
Reply With Quote
 
 
 
 
Carl Banks
Guest
Posts: n/a
 
      04-07-2009
On Apr 7, 12:01*am, k3xji <(E-Mail Removed)> wrote:
> When I run the following function, I seem to have a mem leak, a 20 mb
> of memory
> is allocated and is not freed. Here is the code I run:
>
> >>> import esauth
> >>> for i in range(1000000):

>
> ... * * ss = esauth.penc('sumer')
> ...
>
> >>> for i in range(1000000):

>
> ... * * ss = esauth.penc('sumer')
> ...
>
> And here is the penc() function.
>
> static PyObject *
> penc(PyObject *self, PyObject *args)
> {
> * * * * unsigned char *s= NULL;
> * * * * unsigned char *buf = NULL;
> * * * * PyObject * result = NULL;
> * * * * unsigned int v,len,i = 0;
>
> * * * * if (!PyArg_ParseTuple(args, "s#", &s, &len))
> * * * * return NULL;
>
> * * * * buf = strdup(s);
> * * * * if (!buf) {
> * * * * * * * * PyErr_SetString(PyExc_MemoryError,
> * * * * * * * * * * * * "Out of memory: strdup failed");
> * * * * * * * * return NULL;
> * * * * }
>
> * * * * /*string manipulation*/
>
> * * * * result = PyString_FromString(buf);
> * * * * free(buf);
> * * * * return result;
>
> }
>
> Am I doing something wrong?



It might just be an unfortunate case where malloc keeps allocating
memory higher and higher on the heap even though it frees all the
memory. And since it doesn't give it back to the OS, it runs out.

However, Python apparently does leak a reference if passed a Unicode
object; PyArg_ParseTuple automatically creates an encoded string but
never decrefs it. (That might be necessary evil to preserve
compatibility, though. PyString_AS_STRING does it too.)


Carl Banks
 
Reply With Quote
 
 
 
 
k3xji
Guest
Posts: n/a
 
      04-07-2009
Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
the problem resolved. However, I really cannot understand why the
first version does not work. Here is the latest code that has no
problems at all:

static PyObject *
penc(PyObject *self, PyObject *args)
{
PyObject * result = NULL;
unsigned char *s= NULL;
unsigned char *buf = NULL;
unsigned int v,len,i = 0;

if (!PyArg_ParseTuple(args, "s#", &s, &len))
return NULL;

buf = (unsigned char *) PyMem_Malloc(len);
if (buf == NULL) {
PyErr_NoMemory();
return NULL;
}

/* string manipulation. */

result = PyString_FromStringAndSize((char *)buf, len);
PyMem_Free(buf);
return result;
}

 
Reply With Quote
 
MRAB
Guest
Posts: n/a
 
      04-07-2009
k3xji wrote:
> Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
> the problem resolved. However, I really cannot understand why the
> first version does not work. Here is the latest code that has no
> problems at all:
>
> static PyObject *
> penc(PyObject *self, PyObject *args)
> {
> PyObject * result = NULL;
> unsigned char *s= NULL;
> unsigned char *buf = NULL;
> unsigned int v,len,i = 0;
>
> if (!PyArg_ParseTuple(args, "s#", &s, &len))
> return NULL;
>
> buf = (unsigned char *) PyMem_Malloc(len);
> if (buf == NULL) {
> PyErr_NoMemory();
> return NULL;
> }
>
> /* string manipulation. */
>
> result = PyString_FromStringAndSize((char *)buf, len);
> PyMem_Free(buf);
> return result;
> }
>

In general I'd say don't mix your memory allocators. I don't know
whether CPython implements PyMem_Malloc using malloc, but it's better to
stick with CPython's memory allocators when writing for CPython.
 
Reply With Quote
 
John Machin
Guest
Posts: n/a
 
      04-07-2009
On Apr 7, 9:19*pm, MRAB <(E-Mail Removed)> wrote:
> k3xji wrote:
> > Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
> > the problem resolved. However, I really cannot understand why the
> > first version does not work. Here is the latest code that has no
> > problems at all:

>
> > static PyObject *
> > penc(PyObject *self, PyObject *args)
> > {
> > * *PyObject * result = NULL;
> > * *unsigned char *s= NULL;
> > * *unsigned char *buf = NULL;
> > * *unsigned int v,len,i = 0;

>
> > * *if (!PyArg_ParseTuple(args, "s#", &s, &len))
> > * * * * return NULL;

>
> > * *buf = (unsigned char *) PyMem_Malloc(len);
> > * *if (buf == NULL) {
> > * * * * * *PyErr_NoMemory();
> > * * * * * *return NULL;
> > * *}

>
> > * * * * /* string manipulation. */

>
> > * *result = PyString_FromStringAndSize((char *)buf, len);
> > * *PyMem_Free(buf);
> > * *return result;
> > }

>
> In general I'd say don't mix your memory allocators. I don't know
> whether CPython implements PyMem_Malloc using malloc,


The fantastic manual (http://docs.python.org/c-api/
memory.html#overview) says: """the C allocator and the Python memory
manager ... implement different algorithms and operate on different
heaps""".

> but it's better to
> stick with CPython's memory allocators when writing for CPython.


for the reasons given in the last paragraph of the above reference.

HTH,
John
 
Reply With Quote
 
Benjamin Peterson
Guest
Posts: n/a
 
      04-07-2009
Carl Banks <pavlovevidence <at> gmail.com> writes:
> However, Python apparently does leak a reference if passed a Unicode
> object; PyArg_ParseTuple automatically creates an encoded string but
> never decrefs it. (That might be necessary evil to preserve
> compatibility, though. PyString_AS_STRING does it too.)


Unicode objects cache a copy of themselves as default encoded strings. It is
deallocated when the unicode object its self is.




 
Reply With Quote
 
Floris Bruynooghe
Guest
Posts: n/a
 
      04-07-2009
On Apr 7, 2:10*pm, John Machin <(E-Mail Removed)> wrote:
> On Apr 7, 9:19*pm, MRAB <(E-Mail Removed)> wrote:
>
>
>
> > k3xji wrote:
> > > Interestaing I changed malloc()/free() usage with PyMem_xx APIs and
> > > the problem resolved. However, I really cannot understand why the
> > > first version does not work. Here is the latest code that has no
> > > problems at all:

>
> > > static PyObject *
> > > penc(PyObject *self, PyObject *args)
> > > {
> > > * *PyObject * result = NULL;
> > > * *unsigned char *s= NULL;
> > > * *unsigned char *buf = NULL;
> > > * *unsigned int v,len,i = 0;

>
> > > * *if (!PyArg_ParseTuple(args, "s#", &s, &len))
> > > * * * * return NULL;

>
> > > * *buf = (unsigned char *) PyMem_Malloc(len);
> > > * *if (buf == NULL) {
> > > * * * * * *PyErr_NoMemory();
> > > * * * * * *return NULL;
> > > * *}

>
> > > * * * * /* string manipulation. */

>
> > > * *result = PyString_FromStringAndSize((char *)buf, len);
> > > * *PyMem_Free(buf);
> > > * *return result;
> > > }


I assume you're doing a memcpy() somewhere in there... This is also
safer then your first version since the python string can contain an
embeded \0 and the strdup() of the first version would not copy that.
But maybe you're sure your input doesn't have NULLs in them so it
might be fine.

>
> > In general I'd say don't mix your memory allocators. I don't know
> > whether CPython implements PyMem_Malloc using malloc,

>
> The fantastic manual (http://docs.python.org/c-api/
> memory.html#overview) says: """the C allocator and the Python memory
> manager ... implement different algorithms and operate on different
> heaps""".
>
> > but it's better to
> > stick with CPython's memory allocators when writing for CPython.

>
> for the reasons given in the last paragraph of the above reference.


That document explictly says you're allowed to use malloc() and free()
in extensions. There is nothing wrong with allocating things on
different heaps, I've done and seen it many times and never had
trouble.

Why the original problem ocurred I don't understand either tough.

Regards
Floris
 
Reply With Quote
 
Aahz
Guest
Posts: n/a
 
      04-10-2009
In article <(E-Mail Removed)>,
k3xji <(E-Mail Removed)> wrote:
>
>When I run the following function, I seem to have a mem leak, a 20 mb
>of memory
>is allocated and is not freed. Here is the code I run:
>
>>>> import esauth
>>>> for i in range(1000000):

>
>... ss = esauth.penc('sumer')
>...
>>>> for i in range(1000000):

>
>... ss = esauth.penc('sumer')
>...


BTW, note that if you're using Python 2.x, range(1000000) will cause a
"leak" because ints are never freed. Instead, use xrange().
--
Aahz ((E-Mail Removed)) <*> http://www.pythoncraft.com/

Why is this newsgroup different from all other newsgroups?
 
Reply With Quote
 
Hrvoje Niksic
Guest
Posts: n/a
 
      04-10-2009
http://www.velocityreviews.com/forums/(E-Mail Removed) (Aahz) writes:

> BTW, note that if you're using Python 2.x, range(1000000) will cause
> a "leak" because ints are never freed. Instead, use xrange().


Note that using xrange() won't help with that particular problem.
 
Reply With Quote
 
Carl Banks
Guest
Posts: n/a
 
      04-10-2009
On Apr 9, 11:23*pm, Hrvoje Niksic <(E-Mail Removed)> wrote:
> (E-Mail Removed) (Aahz) writes:
> > BTW, note that if you're using Python 2.x, range(1000000) will cause
> > a "leak" because ints are never freed. *Instead, use xrange().

>
> Note that using xrange() won't help with that particular problem.


I think it will because with xrange the integers will not all have to
exist at one time, so Python doesn't have to increase the size of the
integer pool to a million.


Carl Banks
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
limiting memory consumption of Python itself or a dict in a Python program Jonas Maurus Python 2 09-20-2007 09:10 AM
Low power consumption board with memory Duccio VHDL 0 02-25-2006 10:12 PM
How to calculate the CPU time consumption and memory consuption of any python program in Linux Shahriar Shamil Uulu Python 5 12-24-2005 08:29 PM
Calling the C API from Python and Python program from same C API -bidirectional Praveen, Tayal (IE10) Python 0 03-17-2005 06:33 AM
aspnet_wp.exe memory consumption Kiran Kumar ASP .Net 1 07-15-2003 12:08 PM



Advertisments