Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   maximum number of threads (http://www.velocityreviews.com/forums/t398073-maximum-number-of-threads.html)

Paul Sijben 01-10-2007 07:38 AM

maximum number of threads
 
I have a server in Python 2.5 that generates a lot of threads. It is
running on a linux server (Fedora Core 6).

The server quickly runs out of threads.

I am seeing the following error.

File "/home/sijben/ORCA/src/libmercury_mt.py", line 565, in __init__
MercuryObject.__init__(self,mylink)
File "/home/sijben/ORCA/src/libmercury_mt.py", line 223, in __init__
self.start()
File "/usr/local/lib/python2.5/threading.py", line 434, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread


Does anyone know what it going on here and how I can ensure that I have
all the threads I need?

Paul

Gabriel Genellina 01-10-2007 09:57 AM

Re: maximum number of threads
 
At Wednesday 10/1/2007 04:38, Paul Sijben wrote:

>I have a server in Python 2.5 that generates a lot of threads. It is
>running on a linux server (Fedora Core 6).
>The server quickly runs out of threads.
>
> File "/usr/local/lib/python2.5/threading.py", line 434, in start
> _start_new_thread(self.__bootstrap, ())
>error: can't start new thread
>
>Does anyone know what it going on here and how I can ensure that I have
>all the threads I need?


Simply you can't, as you can't have 10000 open files at once.
Computer resources are not infinite.
Do you really need so many threads? Above a certain threshold, the
program total execution time may increase very quickly.


--
Gabriel Genellina
Softlab SRL






__________________________________________________
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!
http://www.yahoo.com.ar/respuestas


Felipe Almeida Lessa 01-10-2007 11:11 AM

Re: maximum number of threads
 
On 1/10/07, Gabriel Genellina <gagsl-py@yahoo.com.ar> wrote:
> At Wednesday 10/1/2007 04:38, Paul Sijben wrote:
> >Does anyone know what it going on here and how I can ensure that I have
> >all the threads I need?

>
> Simply you can't, as you can't have 10000 open files at once.
> Computer resources are not infinite.
> Do you really need so many threads? Above a certain threshold, the
> program total execution time may increase very quickly.


Maybe Stackless could help the OP?
http://www.stackless.com/

--
Felipe.

Paul Sijben 01-10-2007 11:41 AM

Re: maximum number of threads
 
Gabriel Genellina wrote:
>
> Simply you can't, as you can't have 10000 open files at once. Computer
> resources are not infinite.


sure but *how* fast they run out is the issue here

> Do you really need so many threads?


I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

Paul

Paul Sijben 01-10-2007 11:41 AM

Re: maximum number of threads
 
Gabriel Genellina wrote:
>
> Simply you can't, as you can't have 10000 open files at once. Computer
> resources are not infinite.


sure but *how* fast they run out is the issue here

> Do you really need so many threads?


I might be able to do with a few less but I still need many.

I have done a quick test.

on WinXP I can create 1030 threads
on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)

now the 301 is rather low I'd say.

Paul


Paul Sijben 01-10-2007 11:42 AM

Re: maximum number of threads
 
Felipe Almeida Lessa wrote:

> Maybe Stackless could help the OP?
> http://www.stackless.com/
>


thanks I will look into it!

Paul Sijben 01-10-2007 11:42 AM

Re: maximum number of threads
 
Felipe Almeida Lessa wrote:

> Maybe Stackless could help the OP?
> http://www.stackless.com/
>


thanks I will look into it!

Laurent Pointal 01-10-2007 01:49 PM

Re: maximum number of threads
 
Paul Sijben a écrit :
> Gabriel Genellina wrote:
>> Simply you can't, as you can't have 10000 open files at once. Computer
>> resources are not infinite.

>
> sure but *how* fast they run out is the issue here
>
>> Do you really need so many threads?

>
> I might be able to do with a few less but I still need many.
>
> I have done a quick test.
>
> on WinXP I can create 1030 threads
> on Fedora Core 6 I can only create 301 (both python2.4 and 2.5)
>
> now the 301 is rather low I'd say.


This is a system configurable limit (up to a maximum).

See ulimit man pages.

test

ulimit -a

to see what are the current limits, and try with

ulimit -u 2000

to modify the maximum number of user process (AFAIK each thread use a
process entry on Linux)

>
> Paul
>


Felipe Almeida Lessa 01-10-2007 02:11 PM

Re: maximum number of threads
 
On 1/10/07, Laurent Pointal <laurent.pointal@limsi.fr> wrote:
> This is a system configurable limit (up to a maximum).
>
> See ulimit man pages.
>
> test
>
> ulimit -a
>
> to see what are the current limits, and try with
>
> ulimit -u 2000
>
> to modify the maximum number of user process (AFAIK each thread use a
> process entry on Linux)


I don't think it's only this.

---
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
max rt priority (-r) unlimited
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
---

Well, unlimited number user processes. But:

---
$ python test.py
50
100
150
200
250
300
350
Exception raised: can't start new thread

Biggest number of threads: 382
---

The test.py script is attached.

--
Felipe.


William Heymann 01-10-2007 02:43 PM

Re: maximum number of threads
 
On Wednesday 10 January 2007 7:11 am, Felipe Almeida Lessa wrote:
> ---
> $ python test.py
> 50
> 100
> 150
> 200
> 250
> 300
> 350
> Exception raised: can't start new thread
>
> Biggest number of threads: 382
> ---
>
> The test.py script is attached.


So you know I tried this on ubuntu edgy 64bit edition on a dual 2218 opteron
system with 8G of ram and I got

<lots of output>
Exception raised: can't start new thread
Biggest number of threads: 32274


All times are GMT. The time now is 07:16 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.