Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Segmentation faults using threads

Reply
Thread Tools

Segmentation faults using threads

 
 
Mathias
Guest
Posts: n/a
 
      02-13-2007
Dear ng,

I use the thread module (not threading) for a client/server app where I
distribute large amounts of pickled data over ssh tunnels.
Now I get regular Segmentation Faults during high load episodes. I use a
semaphore to have pickle/unpickle run nonthreaded, but I still get
frequent nondeterministic segmentation faults.
Since there is no traceback after a sf, I have no clue what exactly
happened, and debugging a multithreaded app is no fun anyway

Can someone recommend me how to get extra info during such a crash?
Or any other opinion on where the problem might lie?

Thanks a lot,
Mathias
 
Reply With Quote
 
 
 
 
Daniel Nogradi
Guest
Posts: n/a
 
      02-13-2007
> I use the thread module (not threading) for a client/server app where I
> distribute large amounts of pickled data over ssh tunnels.
> Now I get regular Segmentation Faults during high load episodes. I use a
> semaphore to have pickle/unpickle run nonthreaded, but I still get
> frequent nondeterministic segmentation faults.
> Since there is no traceback after a sf, I have no clue what exactly
> happened, and debugging a multithreaded app is no fun anyway
>
> Can someone recommend me how to get extra info during such a crash?
> Or any other opinion on where the problem might lie?



Hi, it would be helpful if you posted a minimalistic code snippet
which showed the problem you describe.

Daniel
 
Reply With Quote
 
 
 
 
Mathias
Guest
Posts: n/a
 
      02-13-2007
> Hi, it would be helpful if you posted a minimalistic code snippet
> which showed the problem you describe.
>
> Daniel


I wish I could! If I knew exactly where the effect takes place I could
probably circumvent it. All I know know is that it happens under high
load and with a lot of waitstates I can reduce the propability of
crashing. So there must be some race condition somewhere I think.

Is there a way to analyze where the crash took place? I guess I can have
a core dumped and somehow analyze it, but that's probably very hard to do.

Would a profiler work which records the function call structure?

Does someone have experience with threading in python - are there
non-threadsafe functions I should know about?

Thanks,
Mathias
 
Reply With Quote
 
John Nagle
Guest
Posts: n/a
 
      02-13-2007
Daniel Nogradi wrote:
>> I use the thread module (not threading) for a client/server app where I
>> distribute large amounts of pickled data over ssh tunnels.


What module are you using for SSH?

What's in your program that isn't pure Python?
The problem is probably in some non-Python component; you shouldn't
be able to force a memory protection error from within Python code.

Also note that the "marshal" module may be unsafe.

John Nagle
 
Reply With Quote
 
Mathias
Guest
Posts: n/a
 
      02-13-2007
John Nagle wrote:
> Daniel Nogradi wrote:
>>> I use the thread module (not threading) for a client/server app where I
>>> distribute large amounts of pickled data over ssh tunnels.

>
> What module are you using for SSH?
>
> What's in your program that isn't pure Python?
> The problem is probably in some non-Python component; you shouldn't
> be able to force a memory protection error from within Python code.
>
> Also note that the "marshal" module may be unsafe.
>
> John Nagle



I'm using os.popen2() to pipe into an ssh session via stdin/stdout.
That's probably not the elegant way...
Other modules: scipy 0.3.2 (with Numeric 24.2) and python 2.4

Does pickle/cPickle count as part of the marshal module?

Mathias
 
Reply With Quote
 
Mathias
Guest
Posts: n/a
 
      02-13-2007
PS: setting sys.setcheckinterval(1) reduces the probablilty of a failure
as well, but definetely at a performance cost.
 
Reply With Quote
 
Hendrik van Rooyen
Guest
Posts: n/a
 
      02-14-2007
"Mathias" <(E-Mail Removed)> wrote:


> Does someone have experience with threading in python - are there
> non-threadsafe functions I should know about?


how do your threads communicate with one another - are there any
globals that are accessed from different threads?

strange this - you should get an exception, not a segment fault if its
in the python bits..

- Hendrik


 
Reply With Quote
 
Mathias
Guest
Posts: n/a
 
      02-14-2007
>
> What module are you using for SSH?
>
> What's in your program that isn't pure Python?
> The problem is probably in some non-Python component; you shouldn't
> be able to force a memory protection error from within Python code.
>


It looks like the error could be in scipy/Numeric, when a large array's
type is changed, like this:

>>> from scipy import *
>>> a=zeros(100000000,'b') #100 MiB
>>> b=a.copy().astype('d') #800 MiB, ok
>>> a=zeros(1000000000,'b') #1GiB
>>> b=a.copy().astype('d') #8GiB, fails with sf

Segmentation fault

if I use zeros directly for allocation of the doubles it works as expected:

>>> from scipy import *
>>> a=zeros(1000000000,'d') #8GiB, fails with python exception

Traceback (most recent call last):
File "<stdin>", line 1, in ?
MemoryError: can't allocate memory for array
>>>


I use python 2.4, but my scipy and Numeric aren't quite up-to-date:
scipy version 0.3.2, Numeric v 24.2
 
Reply With Quote
 
John Nagle
Guest
Posts: n/a
 
      02-14-2007
Mathias wrote:
>>
>> What module are you using for SSH?
>>
>> What's in your program that isn't pure Python?
>> The problem is probably in some non-Python component; you shouldn't
>> be able to force a memory protection error from within Python code.
>>

>
> It looks like the error could be in scipy/Numeric, when a large array's
> type is changed, like this:
>
> >>> from scipy import *
> >>> a=zeros(100000000,'b') #100 MiB
> >>> b=a.copy().astype('d') #800 MiB, ok
> >>> a=zeros(1000000000,'b') #1GiB
> >>> b=a.copy().astype('d') #8GiB, fails with sf

> Segmentation fault
>
> if I use zeros directly for allocation of the doubles it works as expected:
>
> >>> from scipy import *
> >>> a=zeros(1000000000,'d') #8GiB, fails with python exception

> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> MemoryError: can't allocate memory for array
> >>>

>
> I use python 2.4, but my scipy and Numeric aren't quite up-to-date:
> scipy version 0.3.2, Numeric v 24.2


That sounds like the case where the array has to be reallocated from
4-byte floats to 8-byte doubles is being botched.

Take a look at
"http://www.mail-archive.com/(E-Mail Removed)/msg02033.html"

and then at array_cast in arrraymethods.c of scipy. There may be a
reference count bug in that C code. I'm not familiar enough with
Python reference count internals to be sure, though.

John Nagle
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Debugging segmentation faults George Sakkis Python 4 03-08-2007 04:45 AM
Segmentation faults on "new" ZillionDollarSadist C++ 6 01-18-2007 01:50 PM
using exceptions to catch segmentation faults? Digital Puer C++ 18 12-28-2005 01:31 PM
strcat() and segmentation faults. But malloc() already done. Stanley S C Programming 16 12-22-2005 08:43 PM
YAML custom load: Segmentation faults Kaspar Schiess Ruby 2 06-25-2004 01:01 PM



Advertisments