Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Tuple size and memory allocation for embedded Python

Reply
Thread Tools

Tuple size and memory allocation for embedded Python

 
 
Jinming Xu
Guest
Posts: n/a
 
      01-21-2005
Hi Folks,

Python seems unstable, when allocating big memory. For example, the
following C++ code creates a tuple of tuples:

PyObject* arCoord = PyTuple_New(n);
double d = 1.5;
for(int i=0; i<n; i++)
{
PyObject* coord = PyTuple_New(2);
PyTuple_SetItem(coord,0, PyFloat_FromDouble(d));//x
PyTuple_SetItem(coord,1, PyFloat_FromDouble(d));//y
PyTuple_SetItem(arCoord,i, coord);
}

When the n is small, say 100, the code works fine. when n is big, say
10,000, Python has trouble allocating memory, saying:

"Exception exceptions.IndexError: 'tuple index out of range' in 'garbage
collection' ignored
Fatal Python error: unexpected exception during garbage collection
Aborted"

Could anyone please give me some insight or a fix for this?

Thanks in advance for your answer.

Jinming Xu

__________________________________________________ _______________
Is your PC infected? Get a FREE online computer virus scan from McAfeeŽ
Security. http://clinic.mcafee.com/clinic/ibuy...n.asp?cid=3963

 
Reply With Quote
 
 
 
 
Steve Holden
Guest
Posts: n/a
 
      01-21-2005
Jinming Xu wrote:

> Hi Folks,
>
> Python seems unstable, when allocating big memory. For example, the
> following C++ code creates a tuple of tuples:
>
> PyObject* arCoord = PyTuple_New(n);
> double d = 1.5;
> for(int i=0; i<n; i++)
> {
> PyObject* coord = PyTuple_New(2);
> PyTuple_SetItem(coord,0, PyFloat_FromDouble(d));//x
> PyTuple_SetItem(coord,1, PyFloat_FromDouble(d));//y
> PyTuple_SetItem(arCoord,i, coord);
> }
>
> When the n is small, say 100, the code works fine. when n is big, say
> 10,000, Python has trouble allocating memory, saying:
>
> "Exception exceptions.IndexError: 'tuple index out of range' in 'garbage
> collection' ignored
> Fatal Python error: unexpected exception during garbage collection
> Aborted"
>
> Could anyone please give me some insight or a fix for this?
>
> Thanks in advance for your answer.
>

I'm going to guess that the problem is related to incorrect reference
counts. I don't see any IncRefs in there. It seems probable that the
program will work until you make n high enough to trigger a garbage
collection sweep, then memory your program still regards as allocated is
garbage collected by Python and reused. Ugly

Python is pretty stable, so it's usually best to suspect our own code
unless you're heavily into using the C API (which I'm not, so feel free
to ignore me).

regards
Steve
--
Steve Holden http://www.holdenweb.com/
Python Web Programming http://pydish.holdenweb.com/
Holden Web LLC +1 703 861 4237 +1 800 494 3119
 
Reply With Quote
 
 
 
 
Craig Ringer
Guest
Posts: n/a
 
      01-21-2005
On Fri, 2005-01-21 at 17:20 -0500, Steve Holden wrote:
> Jinming Xu wrote:
>
> > Hi Folks,
> >
> > Python seems unstable, when allocating big memory. For example, the
> > following C++ code creates a tuple of tuples:
> >
> > PyObject* arCoord = PyTuple_New(n);
> > double d = 1.5;
> > for(int i=0; i<n; i++)
> > {
> > PyObject* coord = PyTuple_New(2);
> > PyTuple_SetItem(coord,0, PyFloat_FromDouble(d));//x
> > PyTuple_SetItem(coord,1, PyFloat_FromDouble(d));//y
> > PyTuple_SetItem(arCoord,i, coord);
> > }
> >
> > When the n is small, say 100, the code works fine. when n is big, say
> > 10,000, Python has trouble allocating memory, saying:
> >
> > "Exception exceptions.IndexError: 'tuple index out of range' in 'garbage
> > collection' ignored
> > Fatal Python error: unexpected exception during garbage collection
> > Aborted"
> >
> > Could anyone please give me some insight or a fix for this?
> >
> > Thanks in advance for your answer.
> >

> I'm going to guess that the problem is related to incorrect reference
> counts.


It's usually a safe bet, after all. Another biggie is unchecked return
codes leaving the exception state set, though... that can cause _really_
_weird_ problems. ALWAYS check return values.

> I don't see any IncRefs in there.


In this case it looks OK. PyFloat_FromDouble() reuturns a new reference,
as does PyTuple_New(), and PyTuple_SetItem() steals a reference to its
PyObject* argument.

Of course, there could be refcount errors outside the shown code
segment, but in this case I'd say the immediate error will be because of
an unhandled exception. As to why that exception is being thrown....

Also, forget my comment in my last post about not resizing - I'd failed
to notice the initial creation size of the tuple (the creation of which
isn't checked, but would segfault the app on failure).

> Python is pretty stable, so it's usually best to suspect our own code
> unless you're heavily into using the C API (which I'm not, so feel free
> to ignore me).


That's been my experience - stability issues in my Python/C code have
almost always come down to refcounting bugs and/or failing to detect and
handle or propagate an exception.

--
Craig Ringer

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
static memory allocation versus dynamic memory allocation Ken C Programming 24 11-30-2006 12:37 AM
What is the difference between dynamic memory allocation,and stack allocation ? chris C++ 6 10-28-2005 05:27 AM
Easily convert unicode tuple to python string tuple??? Michal Mikolajczyk Python 1 04-20-2004 08:37 PM
Re: Easily convert unicode tuple to python string tuple??? Jeff Epler Python 0 04-20-2004 03:36 PM
Re: Easily convert unicode tuple to python string tuple??? Bill Scherer Python 0 04-20-2004 03:34 PM



Advertisments