Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Architecture ideas for batch processing?

Reply
Thread Tools

Architecture ideas for batch processing?

 
 
Bjorn Borud
Guest
Posts: n/a
 
      06-13-2005
[Harald <(E-Mail Removed)>]
|
| Call the GC explicitly after dismissing any huge object to convince
| it to really obey the MaxHeapFreeRatio (see [2]).

triggering major collections blindly every second is not exactly an
optimal solution to this problem. this is indeed *very* bad advice.

I would recommend reading a bit more about how the JVM you want to use
manages its memory. if you use Sun's JVM for instance you can read:

http://java.sun.com/docs/hotspot/

I would also recommend using jvmstat and visualgc to inspect what your
JVM is doing. after using it for a while you will become more
familiar with the hotspot GC system and you will most likely be able
to spot various problems with heap sizing etc quite fast.

-Bjørn
 
Reply With Quote
 
 
 
 
Chris Uppal
Guest
Posts: n/a
 
      06-13-2005
Bjorn Borud wrote:

> the important part is to
>
> - have proper abstractions so that later you have the freedom
> to choose.
>
> - implement the remote processing when needed, and not start by
> prematurely assuming that it is required.


I agree with your philosophy here, but I think I'd come to the opposite
conclusion.

Presumably there is no compelling /need/ for the batch code to run in the same
JVM as the online code (the two don't need to interact directly). In that case
I'd want to start with the simple and inherently robust architecture of using
separate processes, and take the options of moving them onto separate machines
or of moving them into a shared JVM as and when it became necessary.

-- chris


 
Reply With Quote
 
 
 
 
Bjorn Borud
Guest
Posts: n/a
 
      06-13-2005
["Chris Uppal" <(E-Mail Removed)-THIS.org>]
|
| I agree with your philosophy here, but I think I'd come to the opposite
| conclusion.
|
| Presumably there is no compelling /need/ for the batch code to run
| in the same JVM as the online code (the two don't need to interact
| directly).

the problem is that we can't really know that based on the information
the OP has posted. if we knew exactly what problem he tried to solve
it would be easier to give recommendations.

-Bjørn
 
Reply With Quote
 
Alan Meyer
Guest
Posts: n/a
 
      06-13-2005
Alan Meyer wrote:
> ...
> I'm looking for architecture suggestions for the batch job
> portion.
> ...


Thank you all for your ideas and suggestions. I will follow
up on them.

If anyone else has more ideas, please chime in. I will continue
to follow the thread.

Alan

 
Reply With Quote
 
Chris Uppal
Guest
Posts: n/a
 
      06-14-2005
Bjorn Borud wrote:

> the problem is that we can't really know that based on the information
> the OP has posted. if we knew exactly what problem he tried to solve
> it would be easier to give recommendations.


Agreed.

-- chris



 
Reply With Quote
 
Bjorn Borud
Guest
Posts: n/a
 
      06-14-2005
["HK" <(E-Mail Removed)>]
|
| May I kindly ask you to carefully read my posting and the
| documentation of ConvinceGC before you suggest I am
| a complete idiot?

indeed, I was mistaken. I misread the API and thought the class was
used to call System.gc() at a given interval indefinitely (which would
be a bad idea, and indeed, bad advice).

| The only thing that I can see that may have mislead you
| is a different understanding of what a "huge object" is.

that, and the fact that I misread the API. my apologies.

-Bjørn
 
Reply With Quote
 
HK
Guest
Posts: n/a
 
      06-14-2005
Bjorn Borud wrote:
> [Harald <(E-Mail Removed)>]
> |
> | Call the GC explicitly after dismissing any huge object to convince
> | it to really obey the MaxHeapFreeRatio (see [2]).
>
> triggering major collections blindly every second is not exactly an
> optimal solution to this problem. this is indeed *very* bad advice.


May I kindly ask you to carefully read my posting and the
documentation of ConvinceGC before you suggest I am
a complete idiot?

The only thing that I can see that may have mislead you
is a different understanding of what a "huge object" is.
For me this is one which needs more than 50% of the
allocated memory. If such an object is not needed
anymore, e.g. after a startup phase of a server,
the only way I found to really get rid of it *and* free the
memory for other processes was ConvinceGC. If you have
a better solution, I would be eager to learn about it.



>
> I would recommend reading a bit more about how the JVM you want to use
> manages its memory. if you use Sun's JVM for instance you can read:
>
> http://java.sun.com/docs/hotspot/


Well, guess were I learned about -XX:MaxHeapFreeRatio.

Harald.

 
Reply With Quote
 
Alan Meyer
Guest
Posts: n/a
 
      06-14-2005
Chris Uppal wrote:
> Bjorn Borud wrote:
>
> > the problem is that we can't really know that based on the information
> > the OP has posted. if we knew exactly what problem he tried to solve
> > it would be easier to give recommendations.

>
> Agreed.
>
> -- chris


The points about abstraction are well taken. I will indeed
follow the advice given and design the interfaces to the programs
so they can run as threads of one JVM, in separate JVMs, or on
separate machines with no changes to the internals of the
processing.

I can't say anything about the specifics of the problem I'm
working on because it's a commercial project and the customer
requires confidentiality from the developers.

Speaking generally, I can say that the batch processes prepare
collections of documents for publication. There are user
interactive components to the system, but the basic publication
process is non-interactive. Publishing is done on a scheduled
basis. Documents pass through a series of steps to transform
them to make them usable by end users.

Thanks.

Alan

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
collocated architecture versus distributed architecture apngss@yahoo.com C Programming 3 09-29-2005 07:44 AM
collocated architecture versus distributed architecture apngss@yahoo.com Java 3 09-29-2005 07:44 AM
collocated architecture versus distributed architecture apngss@yahoo.com C++ 4 09-29-2005 07:44 AM
ON Linux Platform: How can we build binaries for another architecture from 0x86 architecture rashmi C Programming 2 07-05-2005 02:31 PM
how can I use a signal defined in one Architecture to another Architecture Muhammad Khan VHDL 4 07-10-2003 06:14 PM



Advertisments