Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > java thread for core 2 due processors

Reply
Thread Tools

java thread for core 2 due processors

 
 
Motaz K. Saad
Guest
Posts: n/a
 
      08-17-2009
Hello,

I am working on file processing (large number of files) which is
totally independent task for each file. I want to speedup processing
using java thread but I do not want overwhelm my processor (core 2 due
processor) with all threads at the same time.

I want to run 2 parallel threads simultaneously and wait until they
finish then run the next 2 threads. furthermore, each thread allocate
large amount of memory so I need to de-allocate the thread after
collecting the result from it.

I would appreciate if anyone direct my to similar set example
I would appreciate guideline and help


Thanks,
--
Motaz K. Saad
 
Reply With Quote
 
 
 
 
Tom Anderson
Guest
Posts: n/a
 
      08-17-2009
On Mon, 17 Aug 2009, Motaz K. Saad wrote:

> I am working on file processing (large number of files) which is totally
> independent task for each file. I want to speedup processing using java
> thread but I do not want overwhelm my processor (core 2 due processor)
> with all threads at the same time.


You'd have to have an awful lot of threads to 'overwhelm' the processor -
hundreds, or perhaps even thousands. There is a chance that it might
overwhelm the disk, though, depending on things like how big your files
are and how the threads access them.

However, running your jobs in 1000 threads won't be any faster than
running them in 10 (although that might be faster than running them in 2,
because it lets you saturate the disk - two threads on two CPUs will leave
either the disk or CPU underutilised at some point, unless readahead and
GC keep them busy), so your conclusion is right - you want to use fewer
threads than you have tasks.

> I want to run 2 parallel threads simultaneously and wait until they
> finish then run the next 2 threads.


No, you want to run two parallel tasks simultaneously and wait until they
finish then run the next two tasks. You don't need to have one thread per
task.

> furthermore, each thread allocate large amount of memory so I need to
> de-allocate the thread after collecting the result from it.


As long as there are no pointers to the allocated objects after the file
is processed, the garbage collector will reclaim it.

> I would appreciate if anyone direct my to similar set example I would
> appreciate guideline and help


You want:

http://java.sun.com/javase/6/docs/ap...orService.html

Your code looks like:

public class FileProcessingTask implements Runnable {
private final File file;
public FileProcessingTask(File file) {
this.file = file;
}
public void run() {
try {
// process file
}
catch (Exception e) {
// log the exception
// close any open files
}
}
}

public class FileProcessingApp {
public static void main(String... args) {
Collection<File> filesToProcess; // initialise this however you like
int numThreads = Runtime.getRuntime().availableProcessors() * 2;
ExecutorService executor = Executors.newFixedThreadPool(numThreads);
for (File file: filesToProcess) {
executor.execute(new FileProcessingTask(file));
}
executor.awaitTermination(Long.MAX_VALUE, TimeUnit.MILLISECONDS);
}
}

tom

--
Brace yourself for an engulfing, cowardly autotroph! I want your
photosynthetic apparatii!
 
Reply With Quote
 
 
 
 
Joshua Cranmer
Guest
Posts: n/a
 
      08-17-2009
Motaz K. Saad wrote:
> I am working on file processing (large number of files) which is
> totally independent task for each file. I want to speedup processing
> using java thread but I do not want overwhelm my processor (core 2 due
> processor) with all threads at the same time.


Depending on what you are exactly doing, it seems to me that doing two
threads at a time will not be maximizing throughput. They could be
spending most of their time waiting around for disk I/O. On modern
processors, one thread-per-core is generally not optimal; ISTR hearing
somewhere that the correct number is about 1.5-2 per core.

> I want to run 2 parallel threads simultaneously and wait until they
> finish then run the next 2 threads. furthermore, each thread allocate
> large amount of memory so I need to de-allocate the thread after
> collecting the result from it.


As long as you watch who holds references to what, the GC will clear
memory up itself. I think it would generally be sufficient to make sure
that you don't keep references to the threads unless necessary--good
programming practices will likely confine the leaks.

> I would appreciate if anyone direct my to similar set example
> I would appreciate guideline and help


The first thing to recommend is that you have a grasp on how to do
concurrent programming. If your tasks do not communicate to each other
or to the main program, you probably don't have any thread-safety issues.

The best way to actually implement this is probably with the new from
Java 5 ExecutorService API, as others have stated.

--
Beware of bugs in the above code; I have only proved it correct, not
tried it. -- Donald E. Knuth
 
Reply With Quote
 
Roedy Green
Guest
Posts: n/a
 
      08-18-2009
On Mon, 17 Aug 2009 19:16:06 +0100, Tom Anderson
<(E-Mail Removed)> wrote, quoted or indirectly quoted someone who
said :

>
>However, running your jobs in 1000 threads won't be any faster than
>running them in 10 (although that might be faster than running them in 2,


The main time large numbers of threads buys you something is web
scraping. Your threads are mostly sitting waiting for socket i/o to
complete.
--
Roedy Green Canadian Mind Products
http://mindprod.com

http://thecovemovie.com : The Cove: a documentary about Japan's secret atrocities against dolphins.
 
Reply With Quote
 
Roedy Green
Guest
Posts: n/a
 
      08-21-2009
On Tue, 18 Aug 2009 15:42:03 +0200, Christian <(E-Mail Removed)> wrote,
quoted or indirectly quoted someone who said :

>isn't that what NIO is for so you need only one thread to read.. and he
>gives off the processing to an ExecutorService..


Yes. I've been told that is a less flexible, more efficient way to do
it.
--
Roedy Green Canadian Mind Products
http://mindprod.com

http://thecovemovie.com : The Cove: a documentary about Japan's secret atrocities against dolphins.
 
Reply With Quote
 
Tom Anderson
Guest
Posts: n/a
 
      08-25-2009
On Sat, 22 Aug 2009, Christian wrote:

> so I assume that if you really have no interaction with the os and
> nothing that could block you. Then number of Threads = CPU cores should
> be fine/perfect.


Close.

If you're making garbage, you actually want slightly fewer threads than
cores, because the collector will need some CPU time every so often. If
you have 4 cores, you want to run about 3.8 threads. Of course, this is
not possible in practice.

Also, bear in mind that 'no interaction with the OS' includes 'not using
any virtual memory' (in the sense of 'memory which is paged out') - a
thread which accesses a memory location which is paged out will trigger
disk IO, which will cause it to block. If your java processes fits
entirely in physical memory, this is not an issue.

tom

--
Information is not knowledge. -- Albert Einstein
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Core i3-2120 and Core i3-2100 Processors Ian Front Page News 0 04-21-2011 09:40 AM
Intel Core i5-750 and Core i7-870 Processors Ian Front Page News 0 09-09-2009 09:26 AM
Tomcat/Java and multi core processors rym3k Java 1 07-05-2007 11:32 PM
tomcat/java and multi-core processors query gavin@my-deja.com Java 1 03-14-2007 03:39 PM
Core Solo & Core Duo are not Core microarchitecture; 65nm Pentium M chips bigal Hardware 0 03-22-2006 11:24 AM



Advertisments