Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Threads not Improving Performance in Program

Reply
Thread Tools

Re: Threads not Improving Performance in Program

 
 
Jean-Paul Calderone
Guest
Posts: n/a
 
      03-19-2009
On Thu, 19 Mar 2009 09:50:51 -0700, Ryan Rosario <(E-Mail Removed)> wrote:
>I have a parser that needs to process 7 million files. After running
>for 2 days, it had only processed 1.5 million. I want this script to
>parse several files at once by using multiple threads: one for each
>file currently being analyzed.


Threads don't magically make a program faster. In Python in particular,
threads won't do much, if anything, to speed up a CPU bound task. Assuming
you have more than one CPU, Python still limits you to one thread executing
Python bytecode at a time. You could try running multiple processes instead,
if there really is more hardware that's sitting idle with your single
threaded version.

Jean-Paul
 
Reply With Quote
 
 
 
 
Ryan Rosario
Guest
Posts: n/a
 
      03-19-2009
On Mar 19, 10:35*am, Jean-Paul Calderone <(E-Mail Removed)> wrote:
> On Thu, 19 Mar 2009 09:50:51 -0700, Ryan Rosario <(E-Mail Removed)> wrote:
> >I have a parser that needs to process 7 million files. After running
> >for 2 days, it had only processed 1.5 million. I want this script to
> >parse several files at once by using multiple threads: one for each
> >file currently being analyzed.

>
> Threads don't magically make a program faster. *In Python in particular,
> threads won't do much, if anything, to speed up a CPU bound task. *Assuming
> you have more than one CPU, Python still limits you to one thread executing
> Python bytecode at a time. *You could try running multiple processes instead,
> if there really is more hardware that's sitting idle with your single
> threaded version.
>
> Jean-Paul


Thank you for your response. I did not realize that. That seems like a
huge limitation for such a great language.
I will look into forking off processes instead of using threads.

R.
 
Reply With Quote
 
 
 
 
Tim Rowe
Guest
Posts: n/a
 
      03-19-2009
> Thank you for your response. I did not realize that. That seems like a
> huge limitation for such a great language.
> I will look into forking off processes instead of using threads.


If that's what you need to do, yes it is. If it isn't, no it's not. No
language is optimum for all possible applications, each one has
different compromises. You've just discovered one of Python's.

--
Tim Rowe
 
Reply With Quote
 
Vijayendra Bapte
Guest
Posts: n/a
 
      03-20-2009
On Mar 20, 4:21*am, Tim Rowe <(E-Mail Removed)> wrote:
> > Thank you for your response. I did not realize that. That seems like a
> > huge limitation for such a great language.
> > I will look into forking off processes instead of using threads.

>
> If that's what you need to do, yes it is. If it isn't, no it's not. No
> language is optimum for all possible applications, each one has
> different compromises. You've just discovered one of Python's.
>
> --
> Tim Rowe


try this: http://docs.python.org/library/multi...ultiprocessing

- Vijayendra
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Threads not Improving Performance in Program Ryan Rosario Python 1 03-19-2009 04:59 PM
Improving performance of code ruds Java 13 04-08-2007 06:05 PM
improving the performance of my PC LouisB Digital Photography 22 10-23-2006 10:11 AM
Improving Performance in ASP.Net Robin ASP .Net 1 03-04-2005 10:16 PM
Improving VPN performance Frank Toner Cisco 4 04-27-2004 04:08 PM



Advertisments