Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > segfault in extension module

Reply
Thread Tools

segfault in extension module

 
 
Nathaniel Echols
Guest
Posts: n/a
 
      12-01-2003

I've written a function in C to perform protein sequence alignment. This
works fine in a standalone C program. I've added the necessary packaging
to use it in Python; it returns three strings and an integer. However, as
soon as the function is complete, I get a segfault and the interpreter
dies.

If I run Python interactively, just calling the function causes a
segfault. If I'm running a script, I can actually print out the return
values (which are what I'd expect - so something's working) but as soon as
the script is done I get the segfault again. I can even call the function
twice, with different arguments - and it works both times. So it appears
that the problem is with tying up loose ends.

How do I determine what is going wrong? I do not get any problem like
this in the C version. I am not using free() anywhere - I will eventually
need to fix this, but I cannot find any place where I might be accessing
unavailable memory. (Adding in free() does not make any difference in the
module, for what it's worth, but I've had some issues with the C program
so I've left it out.)

[I've also tried using PyMem_Malloc instead, throughout the C code.
Doesn't help.]

thanks,
Nat
(please reply directly!)
 
Reply With Quote
 
 
 
 
Martin v. =?iso-8859-15?q?L=F6wis?=
Guest
Posts: n/a
 
      12-01-2003
Nathaniel Echols <(E-Mail Removed)> writes:

> How do I determine what is going wrong?


I recommend to run your code (or the interactive python) in a debugger.
For example, with gdb, you'd get

gdb /usr/bin/python
(gdb) run
>>> import foo

[program crashes]
(gdb) bt

The latter command will give a backtrace, which should tell you where
it crashes. If you don't get enough detail, make sure you compile your
module with debugging information. If you are on Windows, make then
sure that Python is compiled for debugging as well.

Regards,
Martin
 
Reply With Quote
 
 
 
 
Miki Tebeka
Guest
Posts: n/a
 
      12-01-2003
Hello Nat,

> I've written a function in C to perform protein sequence alignment. This
> works fine in a standalone C program. I've added the necessary packaging
> to use it in Python;

Which one? Are you using the C API?

> it returns three strings and an integer. However, as
> soon as the function is complete, I get a segfault and the interpreter
> dies.

Looks like refcount problems, check out
http://www.python.org/doc/current/ext/refcounts.html .

IMO just avoid all this stuff and use SWIG/Boost.Python/Pyrex.

HTH.
Miki
 
Reply With Quote
 
Nathaniel Echols
Guest
Posts: n/a
 
      12-01-2003
> > I've written a function in C to perform protein sequence alignment. This
> > works fine in a standalone C program. I've added the necessary packaging
> > to use it in Python;

> Which one? Are you using the C API?


Yup - I'm reading right out of the manual.

> Looks like refcount problems, check out
> http://www.python.org/doc/current/ext/refcounts.html .


I read this before and couldn't figure out what it meant. This does seem
like it would relate, but I can't figure out what I'm doing incorrectly.
I just have one function which calls a pure C function and returns a
tuple of strings from it. I'm guessing I need to add a Py_INCREF()
somewhere but so far this just makes it segfault sooner. (I'm
not sure what argument to use for Py_INCREF(), either.)

I've looked at several other pages, and they all seem to involve setups
more complicated than what I'm doing. I'm already using Py_BuildValue()
to generate the returned tuple, and my understanding is that this should
avoid major problems. . .

> IMO just avoid all this stuff and use SWIG/Boost.Python/Pyrex.


I'll look at these, but I only have a tiny little bit of code I need to do
this with - I coded it from scratch with the intention of using it this
way, and could have written it in Python if I didn't care about speed.
Would I really benefit from using one of the other methods? The goal here
is explicitly to put only the very time-dependent code in C; everything
else stays in Python.

thanks,
Nat
 
Reply With Quote
 
Nathaniel Echols
Guest
Posts: n/a
 
      12-01-2003
> The latter command will give a backtrace, which should tell you where it
> crashes. If you don't get enough detail, make sure you compile your
> module with debugging information. If you are on Windows, make then sure
> that Python is compiled for debugging as well. Regards, Martin


Okay:

#0 0x420744fe in _int_free () from /lib/tls/libc.so.6
#1 0x420734d6 in free () from /lib/tls/libc.so.6
#2 0x0809dc0d in _PyObject_GC_Del ()
#3 0x080ce86f in PyDict_Next ()
#4 0x080d191d in _PyModule_Clear ()

I guess this makes sense, but I'm still not sure how to fix it. . .
 
Reply With Quote
 
John J. Lee
Guest
Posts: n/a
 
      12-01-2003
Nathaniel Echols <(E-Mail Removed)> writes:
[...]
> > IMO just avoid all this stuff and use SWIG/Boost.Python/Pyrex.

>
> I'll look at these, but I only have a tiny little bit of code I need to do

[...]
> Would I really benefit from using one of the other methods?


Why not take advantage of it? You've already discovered how
hand-writing extensions can be painful. SWIG is probably best for
you. Ignore all the fancy SWIG features, just ask it to wrap the
function, it's very easy.


> The goal here
> is explicitly to put only the very time-dependent code in C; everything
> else stays in Python.


SWIG should be able to do that. There's probably some overhead above
a hand-written wrapper, but it's so trivial to use SWIG in this simple
way that you probably shouldn't begin to worry about that -- it's
unlikely to be a problem.


John
 
Reply With Quote
 
Michael Hudson
Guest
Posts: n/a
 
      12-01-2003
Nathaniel Echols <(E-Mail Removed)> writes:

> > Looks like refcount problems, check out
> > http://www.python.org/doc/current/ext/refcounts.html .

>
> I read this before and couldn't figure out what it meant.


Then I am pretty sure this is your problem

> This does seem like it would relate, but I can't figure out what I'm
> doing incorrectly. I just have one function which calls a pure C
> function and returns a tuple of strings from it. I'm guessing I
> need to add a Py_INCREF() somewhere but so far this just makes it
> segfault sooner. (I'm not sure what argument to use for
> Py_INCREF(), either.)
>
> I've looked at several other pages, and they all seem to involve setups
> more complicated than what I'm doing. I'm already using Py_BuildValue()
> to generate the returned tuple, and my understanding is that this should
> avoid major problems. . .


Post some code.

Cheers,
mwh

--
If Unicode is a horde of zombies with flaming dung sticks,
the hideous intricacies of JIS, Chinese Big-5, Chinese
Traditional, KOI-8, et cetera are at least an army of ogres
with salt and flensing knives. -- Eric S. Raymond, python-dev
 
Reply With Quote
 
Martin v. =?iso-8859-15?q?L=F6wis?=
Guest
Posts: n/a
 
      12-01-2003
Nathaniel Echols <(E-Mail Removed)> writes:

> Okay:
>
> #0 0x420744fe in _int_free () from /lib/tls/libc.so.6
> #1 0x420734d6 in free () from /lib/tls/libc.so.6
> #2 0x0809dc0d in _PyObject_GC_Del ()
> #3 0x080ce86f in PyDict_Next ()
> #4 0x080d191d in _PyModule_Clear ()
>
> I guess this makes sense, but I'm still not sure how to fix it. . .


Ah, tls/libc.so.6. I think you lose, being confronted with a buggy C
library. Try making it not use /lib/tls.

If that does not change the behaviour, you probably have a
ref-counting bug somewhere. Try building a debugging version of
Python.

Regards,
Martin
 
Reply With Quote
 
Nat Echols
Guest
Posts: n/a
 
      12-04-2003
> Post some code.

may god have mercy on my soul:

#include <Python.h>
#include "nw.h"


static PyObject *nw_align (PyObject *self, PyObject *args) {
char *seq1, *seq2, *mfile;
char *out1, *out2, *match;
int penalty, status, score;
PyObject *results;

if (! PyArg_ParseTuple(args, "sssi", &seq1, &seq2, &mfile, &penalty)) {
return NULL;
}

status = nw(seq1, seq2, mfile, penalty, &out1, &out2, &match, &score);

if (status == -1) {
PyErr_NoMemory();
return NULL;
}

results = Py_BuildValue("(sssi)", out1, out2, match, score);
return results;
}

static PyMethodDef nwMethods[] = {
{"align", nw_align, METH_VARARGS,
"Perform Needleman-Wunsch alignment of two protein sequences."},
{NULL, NULL, 0, NULL} /* This is required! No idea why. */
};

void initnw (void) {
(void) Py_InitModule("nw", nwMethods);
}

This is just the wrapper; the function that it calls is defined such:
int nw (char *seq1, char *seq2, char *matrixfile, int penalty,
char **_out1, char **_out2, char **_match, int *_score);
(I can supply this too, but I already know this works fine in a standalone
C program.)

In Python, I simply do this:
(out1, out2, match, score) = nw.align(seq1, seq2, "BLOSUM62", 0)

This is it; should be simple to fix, no?
 
Reply With Quote
 
Michael Hudson
Guest
Posts: n/a
 
      12-04-2003
Nat Echols <(E-Mail Removed)> writes:

> > Post some code.

>
> may god have mercy on my soul:


Well, I can't see anything flagrantly wrong with that.

Does a debugger provide any hints?

Cheers,
mwh

--
[3] Modem speeds being what they are, large .avi files were
generally downloaded to the shell server instead[4].
[4] Where they were usually found by the technical staff, and
burned to CD. -- Carlfish, asr
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
invoking a segfault within a segfault handler - is this undefinedbehavior? Andrey Vul C Programming 8 07-30-2010 02:14 PM
Using exceptions in defined in an extension module inside anotherextension module Floris Bruynooghe Python 1 12-24-2008 10:33 PM
Segfault accessing dictionary in C Python module Mitko Haralanov Python 3 04-22-2008 05:13 PM
Getting module object by name in C extension module Ilariu Raducan Python 2 07-14-2004 10:56 AM
segfault dup'ing string through C++ (via swig) module David Eger Python 0 07-11-2003 01:55 PM



Advertisments