Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > ulimit stack size and python threads

Reply
Thread Tools

ulimit stack size and python threads

 
 
Greg Lindahl
Guest
Posts: n/a
 
      01-08-2009
I figure this is a FAQ, but I can't find it in any FAQs.

I want to limit the stacksize on my server.

If I set it to 8 megs, or unlimited, python is happy.

If I set it to 4 gigabytes, things like yum (which is a python
program) crash creating a thread. This is on an x86_64 linux kernel,
RHEL5, etc etc.

Why is Python overloading the meaning of the ulimit -s like this?
There are plenty of real non-python programs with huge stack usage,
and I'd like my system default stack limit to be less than unlimited
but much larger than Python will allow.

-- greg

 
Reply With Quote
 
 
 
 
MRAB
Guest
Posts: n/a
 
      01-08-2009
Greg Lindahl wrote:
> I figure this is a FAQ, but I can't find it in any FAQs.
>
> I want to limit the stacksize on my server.
>
> If I set it to 8 megs, or unlimited, python is happy.
>
> If I set it to 4 gigabytes, things like yum (which is a python
> program) crash creating a thread. This is on an x86_64 linux kernel,
> RHEL5, etc etc.
>
> Why is Python overloading the meaning of the ulimit -s like this?
> There are plenty of real non-python programs with huge stack usage,
> and I'd like my system default stack limit to be less than unlimited
> but much larger than Python will allow.
>

I'm only guessing, but could it be a 32-bit limit somewhere? Have you
tried, say, 1GB, which would be within a 32-bit limit?
 
Reply With Quote
 
 
 
 
Greg Lindahl
Guest
Posts: n/a
 
      01-08-2009
> I'm only guessing, but could it be a 32-bit limit somewhere? Have you
> tried, say, 1GB, which would be within a 32-bit limit?


Indeed, ulimit -s 1000000 (a bit smaller than 1 GB) does work, but it
doesn't solve my problem, since I want to set the limit higher than 1
GB.

-- greg
 
Reply With Quote
 
MRAB
Guest
Posts: n/a
 
      01-08-2009
Greg Lindahl wrote:
>> I'm only guessing, but could it be a 32-bit limit somewhere? Have you
>> tried, say, 1GB, which would be within a 32-bit limit?

>
> Indeed, ulimit -s 1000000 (a bit smaller than 1 GB) does work, but it
> doesn't solve my problem, since I want to set the limit higher than 1
> GB.
>

How much higher? You could try just under 4GB (unsigned 32-bit) and just
under 2GB (signed 32-bit).
 
Reply With Quote
 
Martin v. L÷wis
Guest
Posts: n/a
 
      01-08-2009
> Why is Python overloading the meaning of the ulimit -s like this?

Why do you think Python is overloading the meaning of that? I ensure
you it isn't - it doesn't actively care what the limits are.

Regards,
Martin
 
Reply With Quote
 
Greg Lindahl
Guest
Posts: n/a
 
      01-08-2009
> How much higher? You could try just under 4GB (unsigned 32-bit) and just
> under 2GB (signed 32-bit).


I'd like to set it to be about 1/2 the memory size of my server, which
happens to end up being 4 gbytes. And no, slightly less than 4 gb
doesn't work.

But even if that worked, I'd be worried that python is doing something
bad with the ulimit -s value under the covers.

-- greg
 
Reply With Quote
 
Martin v. L÷wis
Guest
Posts: n/a
 
      01-08-2009
> But even if that worked, I'd be worried that python is doing something
> bad with the ulimit -s value under the covers.


Again: it definitely isn't.

Regards,
Martin
 
Reply With Quote
 
Andrew MacIntyre
Guest
Posts: n/a
 
      01-08-2009
Greg Lindahl wrote:
> I figure this is a FAQ, but I can't find it in any FAQs.
>
> I want to limit the stacksize on my server.
>
> If I set it to 8 megs, or unlimited, python is happy.
>
> If I set it to 4 gigabytes, things like yum (which is a python
> program) crash creating a thread. This is on an x86_64 linux kernel,
> RHEL5, etc etc.
>
> Why is Python overloading the meaning of the ulimit -s like this?
> There are plenty of real non-python programs with huge stack usage,
> and I'd like my system default stack limit to be less than unlimited
> but much larger than Python will allow.


The Python interpreter itself (absent a call to resource.setrlimit())
does nothing with resource limits - it just uses the environment it is
loaded into.

In the absence of effective alternative solutions, it may be possible to
achieve the effect you desire by calling resource.setrlimit() in your
Python installation's site.py, effectively over-riding the system
default.

--
-------------------------------------------------------------------------
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: http://www.velocityreviews.com/forums/(E-Mail Removed) (pref) | Snail: PO Box 370
(E-Mail Removed) (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia
 
Reply With Quote
 
Greg Lindahl
Guest
Posts: n/a
 
      01-08-2009
> Why do you think Python is overloading the meaning of that? I ensure
> you it isn't - it doesn't actively care what the limits are.


Always crashing because I asked the OS to please not allow a process
to grow too big is what I call overloading the meaning of ulimit -s.
It's quite surprising. Not to mention the poor error message.

-- greg

 
Reply With Quote
 
Greg Lindahl
Guest
Posts: n/a
 
      01-09-2009
I see. I should be blaming the default behavior of pthreads. I did
work on a OpenMP library once, and we worked around this problem, plus
we gave good error messages. Given the number of HPC sites which use
Python, I'd think that Python would have grown similar features. (HPC
sites are more likely to have intermediate-sized stack limits due to
use of Fortran.)

-- greg
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
C/C++ compilers have one stack for local variables and return addresses and then another stack for array allocations on the stack. Casey Hawthorne C Programming 3 11-01-2009 08:23 PM
ulimit on open sockets ? Maxim Veksler Python 6 04-14-2007 03:39 PM
"stack level too deep"... because Threads keep their "starting" stack Sam Roberts Ruby 1 02-11-2005 04:25 AM
Stack size and bypassing ulimit Kalaky Ruby 0 10-26-2004 01:27 AM
"ulimit -s" has no effect? Maciej Kalisiak Python 16 02-12-2004 06:51 AM



Advertisments