Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Threads, waiting for last one to finish

Reply
Thread Tools

Threads, waiting for last one to finish

 
 
Roedy Green
Guest
Posts: n/a
 
      01-14-2013
On Sun, 13 Jan 2013 10:53:58 -0400, Arved Sandstrom
<(E-Mail Removed)> wrote, quoted or indirectly quoted someone
who said :

>I have difficulty seeing how you wouldn't be able
>to figure out how many threads you propose to start.


I easily do it, it is just that in my case it takes an extra pass.
--
Roedy Green Canadian Mind Products http://mindprod.com
Students who hire or con others to do their homework are as foolish
as couch potatoes who hire others to go to the gym for them.
 
Reply With Quote
 
 
 
 
Arved Sandstrom
Guest
Posts: n/a
 
      01-14-2013
On 01/13/2013 11:26 PM, Arne Vajh°j wrote:
> On 1/13/2013 10:18 PM, Roedy Green wrote:
>> On Sun, 13 Jan 2013 10:53:58 -0400, Arved Sandstrom
>> <(E-Mail Removed)> wrote, quoted or indirectly quoted someone
>> who said :
>>> As for threads taking different times, and "hanging around", I don't see
>>> the problem. If each individual thread does resource cleanup - like
>>> connections - before calling await(), what is the issue with a thread
>>> doing basically nothing?

>>
>> does not a thread have 1 MB + overhead just for existing?

>
> Yes, no, it depends.
>
> It is implementation dependent so it can and do differ between
> different vendors and platforms.
>
> Some may provide an option for changing the default.
>
> I have once read that SUN/Oracle Java on Windows and Linux uses
> 320 KB and 1024KB as default on 32 and 64 bit respectively.
>
> SUN/Oracle Java do provide the -Xss option to change the
> default.
>
> Arne
>

I use the -Xss on occasion for some apps, sometimes it's part of their
tuning docs. Usually I don't worry about thread stack size, not if I've
got a relatively small number (25 certainly being small) of threads.

Not when I need GB of memory for heap, and hundreds of megs for perm gen.

My approach here is my usual approach - go for a simple design, and if a
quick gut-check of performance implications raises no red flags then go
with that for implementation.

AHS
 
Reply With Quote
 
 
 
 
Knute Johnson
Guest
Posts: n/a
 
      01-14-2013
On 1/13/2013 10:38 AM, Robert Klemme wrote:
> On 12.01.2013 06:19, Knute Johnson wrote:
>> On 1/11/2013 1:56 PM, Roedy Green wrote:
>>> I have 25 threads that start at once. I need to wait until the last
>>> one completes. Is there a better way to handle that than using a
>>> ThreadPoolExecutor which seems overkill.
>>>

>>
>> You can always join() them.

>
> I am surprised that this got just one mention so far.


Me too .

--

Knute Johnson
 
Reply With Quote
 
Daniel Pitts
Guest
Posts: n/a
 
      01-14-2013
On 1/13/13 7:35 PM, Arne Vajh°j wrote:
> On 1/13/2013 2:39 AM, Kevin McMurtrie wrote:
>> In article <kcq2ap$m4o$(E-Mail Removed)>,
>> markspace <(E-Mail Removed)> wrote:
>>> On 1/11/2013 1:56 PM, Roedy Green wrote:
>>>> I have 25 threads that start at once. I need to wait until the last
>>>> one completes. Is there a better way to handle that than using a
>>>> ThreadPoolExecutor which seems overkill.
>>>>
>>>
>>> The Executors class has convenience methods to make thread pools easier
>>> to use.
>>>
>>> ExecutorService es = Executors.newFixedThreadPool( 25 );
>>> // submit jobs here
>>> es.shutdown();
>>>
>>> The shutdown() will wait for all jobs to finish. I don't think you can
>>> get easier than that. It's two lines of code!

>>
>> The Executor classes are horribly buggy. Pool size adjustment, idle
>> timeouts, and shutdown sequences do NOT work. Some of the features are
>> impossible to implement efficiently so Sun instead chose to implement
>> them incorrectly.
>>
>> For an Executor, it would be more correct to iterate through all the
>> Future or Callable objects returned and ask for their result.

>
>
> Can you be more specific about when shutdown and await does
> not work?
>
> Arne
>
>

I am also interested. However, I will say that I would have used the
ExecutorService and simply used the Future objects to wait for
completion of all tasks. This has the added benefit that you can
actually process the results of the Callables in a single thread, rather
than have them try to update some shared data structure. Using the
Future result is inherently easier to get correct.
 
Reply With Quote
 
Arne Vajh°j
Guest
Posts: n/a
 
      01-15-2013
On 1/14/2013 4:51 AM, Arved Sandstrom wrote:
> On 01/13/2013 11:26 PM, Arne Vajh°j wrote:
>> On 1/13/2013 10:18 PM, Roedy Green wrote:
>>> On Sun, 13 Jan 2013 10:53:58 -0400, Arved Sandstrom
>>> <(E-Mail Removed)> wrote, quoted or indirectly quoted someone
>>> who said :
>>>> As for threads taking different times, and "hanging around", I don't
>>>> see
>>>> the problem. If each individual thread does resource cleanup - like
>>>> connections - before calling await(), what is the issue with a thread
>>>> doing basically nothing?
>>>
>>> does not a thread have 1 MB + overhead just for existing?

>>
>> Yes, no, it depends.
>>
>> It is implementation dependent so it can and do differ between
>> different vendors and platforms.
>>
>> Some may provide an option for changing the default.
>>
>> I have once read that SUN/Oracle Java on Windows and Linux uses
>> 320 KB and 1024KB as default on 32 and 64 bit respectively.
>>
>> SUN/Oracle Java do provide the -Xss option to change the
>> default.
>>

> I use the -Xss on occasion for some apps, sometimes it's part of their
> tuning docs. Usually I don't worry about thread stack size, not if I've
> got a relatively small number (25 certainly being small) of threads.
>
> Not when I need GB of memory for heap, and hundreds of megs for perm gen.
>
> My approach here is my usual approach - go for a simple design, and if a
> quick gut-check of performance implications raises no red flags then go
> with that for implementation.


I think I have only used it once and that was a demo to prove that
Tomcat could handle 1000 executing but waiting requests in parallel.

But Roedy seemed concerned and the option is there.

Arne


 
Reply With Quote
 
Kevin McMurtrie
Guest
Posts: n/a
 
      01-15-2013
In article <50f37d14$0$293$(E-Mail Removed)>,
Arne Vajh°j <(E-Mail Removed)> wrote:

> On 1/13/2013 2:39 AM, Kevin McMurtrie wrote:
> > In article <kcq2ap$m4o$(E-Mail Removed)>,
> > markspace <(E-Mail Removed)> wrote:
> >> On 1/11/2013 1:56 PM, Roedy Green wrote:
> >>> I have 25 threads that start at once. I need to wait until the last
> >>> one completes. Is there a better way to handle that than using a
> >>> ThreadPoolExecutor which seems overkill.
> >>>
> >>
> >> The Executors class has convenience methods to make thread pools easier
> >> to use.
> >>
> >> ExecutorService es = Executors.newFixedThreadPool( 25 );
> >> // submit jobs here
> >> es.shutdown();
> >>
> >> The shutdown() will wait for all jobs to finish. I don't think you can
> >> get easier than that. It's two lines of code!

> >
> > The Executor classes are horribly buggy. Pool size adjustment, idle
> > timeouts, and shutdown sequences do NOT work. Some of the features are
> > impossible to implement efficiently so Sun instead chose to implement
> > them incorrectly.
> >
> > For an Executor, it would be more correct to iterate through all the
> > Future or Callable objects returned and ask for their result.

>
>
> Can you be more specific about when shutdown and await does
> not work?
>
> Arne


There are race conditions in determining the predicted thread states.
There's simply no way to check the work queue plus all of the states of
all of the threads in an atomic manner without locking or using a master
thread. Sun's implementation has almost no locking and no master thread
so it makes mistakes while threads are transitioning states. The
variable sized thread pool is the worst (glitches maybe 10% of the
time!) but the others have rare glitches too. These glitches are fixed
when a new task is queued so a steady stream of tasks appears to work
correctly. What happened with the very last task is a gamble. That's
why you shouldn't use shutdown() to wait for completion.

I'm not going to walk through the code here. It's awful stuff that's
difficult to follow and it changes a little with each JVM release. You
can step through it in a debugger and see that there's some wishful
thinking in there. Careful placement of breakpoints can produce a fault
that doesn't recover when all breakpoints are removed.
--
I will not see posts from Google because I must filter them as spam
 
Reply With Quote
 
Arved Sandstrom
Guest
Posts: n/a
 
      01-15-2013
On 01/13/2013 02:38 PM, Robert Klemme wrote:
> On 12.01.2013 06:19, Knute Johnson wrote:
>> On 1/11/2013 1:56 PM, Roedy Green wrote:
>>> I have 25 threads that start at once. I need to wait until the last
>>> one completes. Is there a better way to handle that than using a
>>> ThreadPoolExecutor which seems overkill.
>>>

>>
>> You can always join() them.

>
> I am surprised that this got just one mention so far. In absence of
> Executor (which has some issues, as has been mentioned) this is the most
> straightforward way to do it.
>
> // untested and from memory
> final Thread[] threads = new Thread[15];
>
> for ( int i = 0; i < threads.length; ++i ) {
> final Thread th = new Thread(...);
> th.start();
> threads[i] = th;
> }
>
> for ( final Thread th : threads ) {
> th.join();
> }
>
> Kind regards
>
> robert
>

You're both right, of course. One of the hazards - speaking for myself -
of seizing on java.util.concurrent and reading JCIP through several
times is that one may tend to forget the original Java concurrency
constructs.

I think join() satisfies Roedy's problem statement the best; in any case
the latches and barriers have different endgame behaviour.

AHS
 
Reply With Quote
 
markspace
Guest
Posts: n/a
 
      01-15-2013
On 1/14/2013 10:29 PM, Kevin McMurtrie wrote:

> variable sized thread pool is the worst (glitches maybe 10% of the
> time!) but the others have rare glitches too. These glitches are fixed
> when a new task is queued so a steady stream of tasks appears to work
> correctly. What happened with the very last task is a gamble. That's
> why you shouldn't use shutdown() to wait for completion.
>
> I'm not going to walk through the code here. It's awful stuff that's



Hmm, could you produce a code example that glitches 10% of the time?
Otherwise I'm afraid that I'll have to file your claims under "internet
b.s." There's no way I'm going to believe that Oracle's code is that
buggy just on hearsay.



 
Reply With Quote
 
Kevin McMurtrie
Guest
Posts: n/a
 
      01-16-2013
In article <kd3vjc$cat$(E-Mail Removed)>,
markspace <(E-Mail Removed)> wrote:

> On 1/14/2013 10:29 PM, Kevin McMurtrie wrote:
>
> > variable sized thread pool is the worst (glitches maybe 10% of the
> > time!) but the others have rare glitches too. These glitches are fixed
> > when a new task is queued so a steady stream of tasks appears to work
> > correctly. What happened with the very last task is a gamble. That's
> > why you shouldn't use shutdown() to wait for completion.
> >
> > I'm not going to walk through the code here. It's awful stuff that's

>
>
> Hmm, could you produce a code example that glitches 10% of the time?
> Otherwise I'm afraid that I'll have to file your claims under "internet
> b.s." There's no way I'm going to believe that Oracle's code is that
> buggy just on hearsay.


I can prove it to you when you're paying me for it. Here are some bugs
with the variable sized thread pool:

With short tasks on a system having many CPU cores, the time it takes a
thread to be seen as idle becomes large compared to the task execution
time. The result is the needless creation of more threads The surge of
extra threads steals CPU time from existing threads, making the problem
worse. You can demonstrate this by feeding a fixed number of rapidly
completing tasks into a thread pool and then checking the pool size. A
common symptom can be seen in Tomcat on very large servers, where the
connection handler pool suddenly jumps from ~100 to thousands of threads.

The idle timeout doesn't work. Threads are used round-robin so they're
never idle as long as tasks keep coming at a rate of timeout/threads.
For a timeout of 60 seconds, 100 tasks per second will keep a pool of at
least 6000 threads alive. Combine this with the previous bug and you
can see that the variable sized pool is very broken.

Scheduled tasks do not expand the pool size so they starve a thread pool
of workers. Read the code for ScheduledThreadPoolExecutor if you'd like.

java.util.concurrent.Delayed is defined in relative time and must
implement Comparable<Delayed>. It's impossible to sort tasks using
relative time so Sun hacked it. Look at
java.util.concurrent.ScheduledThreadPoolExecutor.S cheduledFutureTask.comp
areTo(Delayed). Any implementation other than
ScheduledFutureCallableLink does not work reliably.

Fixed size pools have some problems too but they're less frequent and
unobtrusive. You'd probably never hit them unless you're using
shutdown() rather than Future.get() to wait for task completion.
--
I will not see posts from Google because I must filter them as spam
 
Reply With Quote
 
Kevin McMurtrie
Guest
Posts: n/a
 
      01-19-2013
In article <kd3vjc$cat$(E-Mail Removed)>,
markspace <(E-Mail Removed)> wrote:

> On 1/14/2013 10:29 PM, Kevin McMurtrie wrote:
>
> > variable sized thread pool is the worst (glitches maybe 10% of the
> > time!) but the others have rare glitches too. These glitches are fixed
> > when a new task is queued so a steady stream of tasks appears to work
> > correctly. What happened with the very last task is a gamble. That's
> > why you shouldn't use shutdown() to wait for completion.
> >
> > I'm not going to walk through the code here. It's awful stuff that's

>
>
> Hmm, could you produce a code example that glitches 10% of the time?
> Otherwise I'm afraid that I'll have to file your claims under "internet
> b.s." There's no way I'm going to believe that Oracle's code is that
> buggy just on hearsay.


I hit another one today and it's directly related to the original post.
The shutdown() method causes a spin loop bug in
ThreadPoolExecutor.getTask().

This is wrong:
if (state == SHUTDOWN) // Help drain queue
r = workQueue.poll();

It should be:
if (state == SHUTDOWN) // Help drain queue
return workQueue.poll();


This was very frustrating because CPU was at 400% but it wouldn't show
up in any Java profiler. I had to pause all threads and start them up
one at a time.

shutdownNow() doesn't cause the bug.
--
I will not see posts from Google because I must filter them as spam
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
waiting for multiple child processes to finish Martin DeMello Ruby 1 11-13-2006 02:30 PM
Print digital photos online - 9 cents per print, plus 10% off. All sizes from 4x6 up to 30x60. Same price for Matte finish and glossy finish nathan_usny Digital Photography 2 09-12-2005 11:30 PM
Print digital photos online - 9 cents per print, plus 10% off. All sizes from 4x6 up to 30x60. Same price for Matte finish and glossy finish nathan_usny Digital Photography 0 09-12-2005 06:09 PM
waiting for external process to finish (using Runtime and Thread classes) brownjenkn@aol.com Java 1 04-22-2005 07:55 PM
Waiting for processes to finish under Solaris Behrang Dadsetan Python 1 07-15-2003 07:56 PM



Advertisments