Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > How to increase number of threads per process?

Reply
Thread Tools

How to increase number of threads per process?

 
 
Ronan Viernes
Guest
Posts: n/a
 
      07-13-2004
Hi,

I have created a python script (see below) to count the maximum number
of threads per process (by starting new threads continuously until it
breaks).

######
#testThread.py
import thread, sys

def main():
print "Main Thread:", thread.get_ident()
count = 0;
try:
while 1:
thread.start_new_thread(test,(`count`,))
count = count + 1;
except:
print "Total Threads:", count
print "Exiting Main Thread:", thread.get_ident()
raise

def test(input=None):
print "count:", thread.get_ident(), input
while 1: #keep thread alive until it breaks
pass

if __name__ == "__main__":
main()

#####

Results:

1. SuSE Professional 7.1 (Kernel 2.4.1, Python 1.5.2
Max Threads = 1024

2. SuSE Professional 7.1 (Kernel 2.4.1, Python 2.0
Max Threads = 1024

3. SuSE Enterprise Server 8.0 (Kernel 2.4.1, Python 1.5.2
Max Threads = 256

4. SuSE Enterprise Server 8.0 (Kernel 2.4.1, Python 2.2
Max Threads = 512

Note:
For all setup, SuSE Linux threads-max=14336 and max_map_count=65536

Questions:
1. How to determine the number of threads? Is it something
configurable?
2. Why do the results above differ in output?
3. How to increase the maximum number of threads per process?


Thanks and hope to hear soon.
Ronan
 
Reply With Quote
 
 
 
 
Steve
Guest
Posts: n/a
 
      07-13-2004
Ronan,

Ronan Viernes wrote:
> Hi,
>
> I have created a python script (see below) to count the maximum number
> of threads per process (by starting new threads continuously until it
> breaks).
>
> ######
> #testThread.py
> import thread, sys
>
> def main():
> print "Main Thread:", thread.get_ident()
> count = 0;
> try:
> while 1:
> thread.start_new_thread(test,(`count`,))
> count = count + 1;
> except:
> print "Total Threads:", count
> print "Exiting Main Thread:", thread.get_ident()
> raise
>
> def test(input=None):
> print "count:", thread.get_ident(), input
> while 1: #keep thread alive until it breaks
> pass
>
> if __name__ == "__main__":
> main()
>
> #####
>
> Results:
>
> 1. SuSE Professional 7.1 (Kernel 2.4.1, Python 1.5.2
> Max Threads = 1024
>
> 2. SuSE Professional 7.1 (Kernel 2.4.1, Python 2.0
> Max Threads = 1024
>
> 3. SuSE Enterprise Server 8.0 (Kernel 2.4.1, Python 1.5.2
> Max Threads = 256
>
> 4. SuSE Enterprise Server 8.0 (Kernel 2.4.1, Python 2.2
> Max Threads = 512
>
> Note:
> For all setup, SuSE Linux threads-max=14336 and max_map_count=65536
>
> Questions:
> 1. How to determine the number of threads? Is it something
> configurable?
> 2. Why do the results above differ in output?
> 3. How to increase the maximum number of threads per process?
>


I'm not sure if this is configurable but I was testing out something
similar today (on fedora core 1). I discovered that a structure such as
the following works with any number of thread (I tried it with 2000
threads just now). The main thing I did was introducing a
'time.sleep(0.01)' in the loop that waits for the worker threads to join
the main thread, and adding a try..except clause around where you start
the threads. Basically an exception is never thrown because I run 10
threads at one go, and then sleep for a fraction of a second (it's
usually a good idea to sleep/yield in busy loops). I think the main
thing here is the fact that it takes 10 threads at a time and then gives
it a pause.

Please try this out and let me know what you find.

class A:
cache = {}
counter = 0
def local(self):
A.cache[A.counter*2] = A.counter
A.counter += 1

def print_it(self):
print A.cache, len(A.cache), A.counter


def test():
count = 2000 # I'm pretty sure a larger number should work too

a = A()
threadList = []
for i in xrange(count):
try:
t = threading.Thread(target=a.local, args=())
threadList.append(t)
t.start()

# Adding the following if statement allowed running more
than 256 threads in one go
# Previously it would run only 256 threads if this clause
wasn't there.
if i % 10 == 0 and i != 0:
# setting sys.setcheckinterval to 1 didn't help. Let's try
# voluntarily giving up some cycles -- this should give
# other threads a chance to run
time.sleep(0.01) # this works with time.sleep(0.005)
as well
# end if
except:
continue
# end try
# end for

# Wait for threads to join the main thread
while threading.activeCount()>1:
time.sleep(1)

a.print_it()

test()


>
> Thanks and hope to hear soon.
> Ronan


Steve

 
Reply With Quote
 
 
 
 
Ronan Viernes
Guest
Posts: n/a
 
      07-14-2004
Hi Steve,

Thanks for the info. Unfortunately, adding the sleep to my previous
example (modified version below) did not alter the results. Please
take note that I am using thread library (and not threading). The
sample code below is simplified version of our system and of course
each of the thread has other tasks to do rather than staying idle in
there.

What really puzzles me now is, if I change the environment (OS and
Python Versions) accordingly it will give different results. This made
me believe that it is somewhat 'configurable'.

Anyway, I really appreciate the help. Let's wait what the others have
to say.

> > ######
> > #testThread.py
> > import thread, sys
> >
> > def main():
> > print "Main Thread:", thread.get_ident()
> > count = 0;
> > try:
> > while 1:
> > thread.start_new_thread(test,(`count`,))
> > count = count + 1;
> > if count % 10 == 0 and count != 0:
> > time.sleep(0.01)
> > except:
> > print "Total Threads:", count
> > print "Exiting Main Thread:", thread.get_ident()
> > raise
> >
> > def test(input=None):
> > print "count:", thread.get_ident(), input
> > while 1: #keep thread alive until it breaks
> > pass
> >
> > if __name__ == "__main__":
> > main()
> >
> > #####
> >
> > Results:
> >
> > 1. SuSE Professional 7.1 (Kernel 2.4.1, Python 1.5.2
> > Max Threads = 1024
> >
> > 2. SuSE Professional 7.1 (Kernel 2.4.1, Python 2.0
> > Max Threads = 1024
> >
> > 3. SuSE Enterprise Server 8.0 (Kernel 2.4.1, Python 1.5.2
> > Max Threads = 256
> >
> > 4. SuSE Enterprise Server 8.0 (Kernel 2.4.1, Python 2.2
> > Max Threads = 512
> >
> > Note:
> > For all setup, SuSE Linux threads-max=14336 and max_map_count=65536
> >
> > Questions:
> > 1. How to determine the number of threads? Is it something
> > configurable?
> > 2. Why do the results above differ in output?
> > 3. How to increase the maximum number of threads per process?
> >


Regards,
Ronan
 
Reply With Quote
 
John Lenton
Guest
Posts: n/a
 
      07-14-2004
On 13 Jul 2004 19:27:21 -0700, Ronan Viernes <(E-Mail Removed)> wrote:
> Hi Steve,
>
> Thanks for the info. Unfortunately, adding the sleep to my previous
> example (modified version below) did not alter the results. Please
> take note that I am using thread library (and not threading). The
> sample code below is simplified version of our system and of course
> each of the thread has other tasks to do rather than staying idle in
> there.
>
> What really puzzles me now is, if I change the environment (OS and
> Python Versions) accordingly it will give different results. This made
> me believe that it is somewhat 'configurable'.


as far as I know, it's the way your glibc is compiled that makes the difference.

--
John Lenton ((E-Mail Removed)) -- Random fortune:
bash: fortune: command not found
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
Per packet vs per flow routing R Siffredi Cisco 8 03-24-2005 06:33 PM
Per-to-Per is OK but no ICS Jens Mander Wireless Networking 4 01-23-2005 10:24 PM
Re: How to increase number of threads per process? John Lenton Python 0 07-14-2004 11:22 AM
RE: How to increase number of threads per process? Ronnan-Ian Viernes Python 0 07-14-2004 03:29 AM



Advertisments