Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Re: How can you make idle processors pick up java work?

Reply
Thread Tools

Re: How can you make idle processors pick up java work?

 
 
Patricia Shanahan
Guest
Posts: n/a
 
      08-01-2012
From: Patricia Shanahan <(E-Mail Removed)>

On 7/31/2012 1:07 PM, qwertmonkey wrote: ...
> Please, let me know if you have a better idea of how to read very large

text
> files in a faster way (while using a regular/sub-optimal box) ~

....

It sounds as though your throughput is limited by disk read head time -
splitting across physical disks increases that resource.

You *may* be able to improve the efficiency of one drive by using NIO
non-blocking reads to keep more disk prefetch reads going at once. Given a pool
of reads, the drive can optimize their order to reduce total head movement, and
therefore time the read head wastes not actually doing a transfer.

Patricia

--- BBBS/Li6 v4.10 Dada-1
* Origin: Prism bbs (1:261/3
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24
 
Reply With Quote
 
 
 
 
Lew
Guest
Posts: n/a
 
      08-01-2012
To: Patricia Shanahan
From: Lew <(E-Mail Removed)>

Patricia Shanahan wrote:
> qwertmonkey wrote:
> ...
>
>> Please, let me know if you have a better idea of how
>> to read very large text
>> files in a faster way (while using a regular/sub-optimal box) ~

>
> ...


Another approach is to farm out one file per thread, rather than multiple
threads per file.

Part of the game of concurrency is to figure out the right size for a chunk.

Your problem also seems amenable to a map-reduce approach.

> It sounds as though your throughput is limited by disk read head time -
> splitting across physical disks increases that resource.
>
> You *may* be able to improve the efficiency of one drive by using NIO
> non-blocking reads to keep more disk prefetch reads going at once. Given
> a pool of reads, the drive can optimize their order to reduce total head
> movement, and therefore time the read head wastes not actually doing a
> transfer.


You can also perhaps improve efficiency with the one thread-per-file approach.

I/O in each thread should be bursty, especially if you use
'BufferedReader' with large buffers. Each thread's I/O time
is potentially CPU time for other threads. Also, multiple threads' I/O might
bunch up for elevator seeking as Patricia describes.

However, threads hit a limit of usefulness if you get too many. Fortunately
that limit should be higher than that imposed by your I/O subsystem. You
control this by setting your thread pool's maximum count and other control
factors appropriately.

For the kind of situation you describe I'd predict the optimum maximum to be
somewhere around two to four times your core count.

I don't know whether NIO or thread-based expansion will help you more. Even the
predictions I am making are implicity heavily burdened with caveats to measure
and take nothing for granted.

I'd likely use a one thread-per-file approach for structural reasons, without
much initial concern for whether it's the best performing. It's a clean
architecture that allows good locality of working variables and a natural path
to a service approach. I wouldn't expect any optimization beyond large buffers
to help individual file performance, so I'd plan for an architecture that
scales well with increased hardware. One file thread per file scales well to
one service call per file, readily distributed across a cluster. It also fits
well with expansions to an I/O subsystem.

Amdahl's Law is your friend if you use it to plan units of work that have
little to no common paths.

--
Lew

--- BBBS/Li6 v4.10 Dada-1
* Origin: Prism bbs (1:261/3
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How can you make idle processors pick up java work? qwertmonkey@syberianoutpost.ru Java 0 07-31-2012 05:30 PM
How can you make idle processors pick up java work? qwertmonkey@syberianoutpost.ru Java 0 07-31-2012 12:30 PM
How can you make idle processors pick up java work? qwertmonkey@syberianoutpost.ru Java 2 07-31-2012 11:41 AM
How can you make idle processors pick up java work? lbrtchx@gemale.com Java 3 07-31-2012 11:16 AM
How can you make idle processors pick up java work? qwertmonkey@syberiaoutpost.ru Java 3 07-31-2012 03:45 AM



Advertisments