Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Permanent Generation and garbage collection

Reply
Thread Tools

Permanent Generation and garbage collection

 
 
Tassilo Horn
Guest
Posts: n/a
 
      08-12-2009
Hi all,

we have some MDA-style library here. Basically, you define some schema
(something like a UML class diagram), and then a code generator creates
bytecode for the classes and associations in memory, and a custom class
loader invokes ClassLoader.defineClass(). Then you can create graphs
(like UML object diagrams) conforming that schema, which instantiates
the generated classes.

Now there's also a model-transformation language, that creates new
graphs conforming to new schemas out of old graphs, and those new
classes are also compiled in memory. Many transformations run in
several steps, and each step compiles new classes.

So it's not seldom, that finally there are 10.000 classes in memory,
which results in an OutOfMemoryError: PermGen space. Increasing PermGen
space using the relevant command line switch works, but isn't there a
better way?

For example in such step-by-step transformations, I'm sure that the
classes created in intermediate steps cannot be used from outside, and
that it's ok to forget the classes defined in step i when finishing step
i+1. For those intermediate classes, I do know the exact prefix of the
fully qualified name all those classes share, so I'm looking for
something like

System.forgetAllClasses(myPrefix);

Any pointers are highly appreciated.

Bye,
Tassilo
 
Reply With Quote
 
 
 
 
Tassilo Horn
Guest
Posts: n/a
 
      08-12-2009
Thomas Pornin <(E-Mail Removed)> writes:

Hi Thomas,

> As far as I know, the JVM may "forget" a class only by forgetting the
> ClassLoader instance which loaded the class, which happens only if
> nobody references the ClassLoader,


Our ClassLoader is a singleton, so it references itself.

> any of its loaded classes, any of the instances created using such
> classes, or any of the static fields declared in said classes.


Well, I think that's the case.

> So if you want to load 10000 classes in a single ClassLoader, then you
> will have to size PermGen such that all 10000 classes fit
> simultaneously.
>
> If you want to forget some classes, then you will have to create
> distinct ClassLoader instances. Maybe each of your defined classes
> could use a specific ClassLoader ?


If you are right, a distinct ClassLoader for each schema would make much
sense.

Thanks for the pointers,
Tassilo
 
Reply With Quote
 
 
 
 
Tassilo Horn
Guest
Posts: n/a
 
      08-12-2009
Tassilo Horn <(E-Mail Removed)> writes:

Hi again,

>> So if you want to load 10000 classes in a single ClassLoader, then
>> you will have to size PermGen such that all 10000 classes fit
>> simultaneously.
>>
>> If you want to forget some classes, then you will have to create
>> distinct ClassLoader instances. Maybe each of your defined classes
>> could use a specific ClassLoader ?

>
> If you are right, a distinct ClassLoader for each schema would make
> much sense.


Ok, now I've done that. Our ClassLoader is now a "multi-singleton",
where the instances are held in a Map from schemaName to ClassLoader.
Whenever I don't need a schema anymore, I release its ClassLoader and
call System.gc(). I track the number of ClassLoaders, and it's at most
3, which is correct.

Unfortunately, I still get the OutOfMemoryError: PermGen, although if
the classes were really freed from the permanent generation, I should
need about half the space I needed before that change.

Is there some way to check, if someone still references the ClassLoader
or any Object of a class loaded by that ClassLoader?

Bye,
Tassilo
 
Reply With Quote
 
Lew
Guest
Posts: n/a
 
      08-12-2009
Tassilo Horn wrote:
> call System.gc().


Generally useless, especially so with respect to the permanent generation.

But going back to your question:
> Increasing PermGen space using the relevant command line switch works,
> but isn't there a better way?


What constitutes "better"? You know you need a lot of perm-gen space, you
allocate it. That seems the epitome of goodness.

-XX:+CMSClassUnloadingEnabled perhaps?

The relevant command-line switches aren't mutually exclusive. You could use
them in combination.

--
Lew
 
Reply With Quote
 
Tassilo Horn
Guest
Posts: n/a
 
      08-12-2009
Lew <(E-Mail Removed)> writes:

Hi Lew,

>> call System.gc().

>
> Generally useless, especially so with respect to the permanent
> generation.


Noted.

> But going back to your question:
>> Increasing PermGen space using the relevant command line switch
>> works, but isn't there a better way?

>
> What constitutes "better"?


No command line switches -- the system arranges all for a user.

> You know you need a lot of perm-gen space, you allocate it. That
> seems the epitome of goodness.


Well, but I don't too much perm-gen space at a given point in time. I
only need a lot if I cannot free it by garbage collecting classes I
don't use anymore.

> -XX:+CMSClassUnloadingEnabled perhaps?


No, it doesn't seem to make a (noticable) difference.

> The relevant command-line switches aren't mutually exclusive. You
> could use them in combination.


Yes, done that. No matter if I add -XX:+CMSClassUnloadingEnabled, I
have to set max perm-gen to the same amount. (Ok, maybe I could drop a
few MBs, but it's not that much that it makes a difference.)

Do you know if Thomas was right with his assumption that if a
ClassLoader has no references anymore, as well as all classes it loaded,
then it should be garbage collected?

Bye,
Tassilo
 
Reply With Quote
 
Roedy Green
Guest
Posts: n/a
 
      08-12-2009
On Wed, 12 Aug 2009 17:04:01 +0200, Tassilo Horn
<(E-Mail Removed)> wrote, quoted or indirectly quoted someone
who said :

>Is there some way to check, if someone still references the ClassLoader
>or any Object of a class loaded by that ClassLoader?


see http://mindprod.com/jgloss/profiler.html
for tools that might help you figure out what is pinning your classes
in RAM.
--
Roedy Green Canadian Mind Products
http://mindprod.com

"You can have quality software, or you can have pointer arithmetic; but you cannot have both at the same time."
~ Bertrand Meyer (born: 1950 age: 59) 1989, creator of design by contract and the Eiffel language.
 
Reply With Quote
 
Tassilo Horn
Guest
Posts: n/a
 
      08-12-2009
Patricia Shanahan <(E-Mail Removed)> writes:

Hi Patricia,

>> Do you know if Thomas was right with his assumption that if a
>> ClassLoader has no references anymore, as well as all classes it
>> loaded, then it should be garbage collected?

> ...
>
> The JLS certainly permits garbage collection in that situation. See
> http://java.sun.com/docs/books/jls/t...tion.html#12.7
> That does not guarantee it will happen.


Thanks.

> I wonder whether it would be possible to use
> java.lang.ref.WeakReference and a ReferenceQueue to see whether any of
> the ClassLoader and Class objects are being freed?


For the custom ClassLoader I do count object creations and finalizations
now. And indeed, after I dropped the last reference to some class
loader (at least in my code, dunno if there are references to it in the
JVM internals), the finalize() method isn't called. So I guess, there
are still references to the classes it loaded, or to objects of those
classes.

So I'll play around with the profilers Roedy links.

Bye,
Tassilo
 
Reply With Quote
 
Lew
Guest
Posts: n/a
 
      08-12-2009
Patricia Shanahan <(E-Mail Removed)> writes:
>> I wonder whether it would be possible to use
>> java.lang.ref.WeakReference and a ReferenceQueue to see whether any of
>> the ClassLoader and Class objects are being freed?


Tassilo Horn wrote:
> For the custom ClassLoader I do count object creations and finalizations
> now. And indeed, after I dropped the last reference to some class
> loader (at least in my code, dunno if there are references to it in the
> JVM internals), the finalize() method isn't called. So I guess, there
> are still references to the classes it loaded, or to objects of those
> classes.


Finalizers make life worse for GC. Patricia's suggestions will allow you to
monitor that things are released without the overhead of finalize().

For one thing, having a non-trivial finalize() guarantees that a dead object
will need a minimum of two collection cycles to release. If you are using
lots of heap with finalizable objects, the collector might not be able to gain
enough in one collection to avoid OOME.

--
Lew
 
Reply With Quote
 
Tassilo Horn
Guest
Posts: n/a
 
      08-12-2009
Lew <(E-Mail Removed)> writes:

Hi Lew,

>>> I wonder whether it would be possible to use
>>> java.lang.ref.WeakReference and a ReferenceQueue to see whether any
>>> of the ClassLoader and Class objects are being freed?

>
> Tassilo Horn wrote:
>> For the custom ClassLoader I do count object creations and
>> finalizations now. And indeed, after I dropped the last reference to
>> some class loader (at least in my code, dunno if there are references
>> to it in the JVM internals), the finalize() method isn't called. So
>> I guess, there are still references to the classes it loaded, or to
>> objects of those classes.

>
> Finalizers make life worse for GC. Patricia's suggestions will allow
> you to monitor that things are released without the overhead of
> finalize().


Ok, I'll have a look at that.

> For one thing, having a non-trivial finalize() guarantees that a dead
> object will need a minimum of two collection cycles to release.


Is that non-trivial and will hinder GC?

@Override
protected void finalize() throws Throwable {
synchronized (this) {
instanceCount--;
}
super.finalize();
}

> If you are using lots of heap with finalizable objects, the collector
> might not be able to gain enough in one collection to avoid OOME.


Only my custom ClassLoader overrides finalize(), and those are only 4
instances at the time I get the OOME with 128MB PermGen space. Each of
them loads about 2000 classes.

Bye,
Tassilo
 
Reply With Quote
 
Joshua Cranmer
Guest
Posts: n/a
 
      08-12-2009
Tassilo Horn wrote:
> For the custom ClassLoader I do count object creations and finalizations
> now. And indeed, after I dropped the last reference to some class
> loader (at least in my code, dunno if there are references to it in the
> JVM internals), the finalize() method isn't called. So I guess, there
> are still references to the classes it loaded, or to objects of those
> classes.


Using finalize() can wreak havoc with garbage collection: at they very
least, it requires two collection attempts to actually clear the memory.
I have to imagine that for the purposes of unloading storage space for
classes, it will require many more, probably more than the garbage
collector will care to attempt in memory pressure situations.

If you play around with java.lang.ref, you can avoid some potential
finalize() problems, although I don't know how well reference queues
work in this regard.

--
Beware of bugs in the above code; I have only proved it correct, not
tried it. -- Donald E. Knuth
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Collection problems (create Collection object, add data to collection, bind collection to datagrid) Řyvind Isaksen ASP .Net 1 05-18-2007 09:24 AM
Templates - Garbage In Garbage Not Out ramiro_b@yahoo.com C++ 1 07-25-2005 04:48 PM
Memory Usage and Garbage Collection Andy ASP .Net 6 03-05-2004 09:08 AM
ASP.NET Session State Server and garbage collection =?Utf-8?B?RnJhbms=?= ASP .Net 5 01-31-2004 03:51 PM
Garbage Collection and Manage Code? Laser Lu ASP .Net 5 01-27-2004 03:48 AM



Advertisments