Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > How can you make idle processors pick up java work?

Reply
Thread Tools

How can you make idle processors pick up java work?

 
 
qwertmonkey
Guest
Posts: n/a
 
      07-31-2012
From: http://www.velocityreviews.com/forums/(E-Mail Removed)

>> Is there a way to make these processors pick up/share work also, or
>> do you have to use some sort of scheduling framework on top of java?


> Use multiple threads?

~
a) I need to actually scan large text files (10+ million lines).
b) On each line there is a NL sentence.
c) That processing should be run only once, but as fast as possible.
~
d) If you go:
d.1) int iPrx = Runtime.getRuntime().availableProcessors();
d.2) count all lines
d.3) split the file in (total lines)/iPrx
d.4) then run iPrx threads (or executable instances using a batch script)
the time you waste on d.2) and d.3) will make all that strat senseless
~
I have no way to influence how those large files are generated
~
e) because of the large sizes of the files you can't even go
~
FIS = new FileInputStream(IFl);
FileChannel IFlChnl = FIS.getChannel();
int iChnlSz = (int)IFlChnl.size();
MappedByteBuffer MptBytBfr = IFlChnl.map(FileChannel.MapMode.READ_ONLY, 0,
iChnlSz);
~
so, apparently, the only option I have is:
~
BfR = Files.newBufferedReader(DirPth, ChrStUTF;
String aSx = BfR.readLine();
while(aSx != null){

aSx = BfR.readLine();
}
~
do you know of a faster way to go about this?
~
lbrtchx

--- BBBS/Li6 v4.10 Dada-1
* Origin: Prism bbs (1:261/3
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24
 
Reply With Quote
 
 
 
 
Patricia Shanahan
Guest
Posts: n/a
 
      07-31-2012
To: qwertmonkey
From: Patricia Shanahan <(E-Mail Removed)>

On 7/30/2012 4:27 PM, (E-Mail Removed) wrote:
>>> Is there a way to make these processors pick up/share work also, or
>>> do you have to use some sort of scheduling framework on top of java?

>
>> Use multiple threads?

> ~
> a) I need to actually scan large text files (10+ million lines).
> b) On each line there is a NL sentence.
> c) That processing should be run only once, but as fast as possible.
> ~
> d) If you go:
> d.1) int iPrx = Runtime.getRuntime().availableProcessors();
> d.2) count all lines
> d.3) split the file in (total lines)/iPrx
> d.4) then run iPrx threads (or executable instances using a batch script)
> the time you waste on d.2) and d.3) will make all that strat senseless


Why worry about splitting by actual line count, rather than by byte position in
file?

Patricia

--- BBBS/Li6 v4.10 Dada-1
* Origin: Prism bbs (1:261/3
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24
 
Reply With Quote
 
 
 
 
David Lamb
Guest
Posts: n/a
 
      07-31-2012
To: qwertmonkey
From: David Lamb <(E-Mail Removed)>

On 30/07/2012 7:27 PM, (E-Mail Removed) wrote:
>>> Is there a way to make these processors pick up/share work also, or
>>> do you have to use some sort of scheduling framework on top of java?

>
>> Use multiple threads?

> ~
> a) I need to actually scan large text files (10+ million lines).
> b) On each line there is a NL sentence.
> c) That processing should be run only once, but as fast as possible.
> ~
> d) If you go:
> d.1) int iPrx = Runtime.getRuntime().availableProcessors();
> d.2) count all lines
> d.3) split the file in (total lines)/iPrx
> d.4) then run iPrx threads (or executable instances using a batch script)
> the time you waste on d.2) and d.3) will make all that strat senseless


How slow is the NL processing? Does it make any sense to read lines in one
thread and pass each off to one of the iPrx-1 other threads that might run on
separate processors?

--- BBBS/Li6 v4.10 Dada-1
* Origin: Prism bbs (1:261/3
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24
 
Reply With Quote
 
Joshua Cranmer
Guest
Posts: n/a
 
      07-31-2012
To: qwertmonkey
From: Joshua Cranmer <(E-Mail Removed)>

[Gah, your newsreader is incapable of threading posts correctly. Please find a
non-broken one.]

On 7/30/2012 7:27 PM, (E-Mail Removed) wrote:
>>> Is there a way to make these processors pick up/share work also, or
>>> do you have to use some sort of scheduling framework on top of java?

>
>> Use multiple threads?

> ~
> a) I need to actually scan large text files (10+ million lines).
> b) On each line there is a NL sentence.
> c) That processing should be run only once, but as fast as possible.


Only 10M-line files?

The easiest way to do this is to just make a ThreadPoolExecutor and have your
main thread dispatch requests as fast as possible to the pool. Or you can do
the work pooling yourself, which may be faster since you're not continually
posting Runnable's, but timing results would be necessary to convince me.

There are other options, but chances are, your disk drive is going to saturate
first (in short, it involves reading non-consecutive pages of the file, which
is generally a recipe for disaster).

--
Beware of bugs in the above code; I have only proved it correct, not tried it.
-- Donald E. Knuth

--- BBBS/Li6 v4.10 Dada-1
* Origin: Prism bbs (1:261/3
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24
 
Reply With Quote
 
John B. Matthews
Guest
Posts: n/a
 
      07-31-2012
To: qwertmonkey
From: "John B. Matthews" <(E-Mail Removed)>

In article <jv759b$s7$(E-Mail Removed)>,
(E-Mail Removed) wrote:

> d.2) count all lines


Maybe ask ProcessBuilder to `wc -l`, or similar?

--
John B. Matthews
trashgod at gmail dot com
<http://sites.google.com/site/drjohnbmatthews>

--- BBBS/Li6 v4.10 Dada-1
* Origin: Prism bbs (1:261/3
--- Synchronet 3.16a-Win32 NewsLink 1.98
Time Warp of the Future BBS - telnet://time.synchro.net:24
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How can you make idle processors pick up java work? qwertmonkey@syberianoutpost.ru Java 0 07-31-2012 05:30 PM
How can you make idle processors pick up java work? qwertmonkey@syberianoutpost.ru Java 0 07-31-2012 12:30 PM
How can you make idle processors pick up java work? qwertmonkey@syberianoutpost.ru Java 2 07-31-2012 11:41 AM
How can you make idle processors pick up java work? lbrtchx@gemale.com Java 3 07-31-2012 11:16 AM
How can you make idle processors pick up java work? qwertmonkey@syberiaoutpost.ru Java 3 07-31-2012 03:45 AM



Advertisments