Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > OutOfMemoryError - how to find root cause

Reply
Thread Tools

OutOfMemoryError - how to find root cause

 
 
kaeli
Guest
Posts: n/a
 
      04-25-2005

Very recently, a web application ( JSP / Iplanet6 / Solaris 2. has begun to
freeze up. Web server logs indicate:
Internal error: Unexpected error condition thrown
java.lang.OutOfMemoryError,no description), stack: java.lang.OutOfMemoryError

This happens intermittently, with no obvious cause, and it just started
happening this morning (no changes have been made to either the server or the
application). It resolves itself and the application resumes just fine. But
it's irritating for pages to freeze up on people.

I *suspect* another application is using too much memory, but can't figure
out if that is the case.
Raising the JVM heap (in server admin GUI) to 256 didn't help. We can't /
don't want to raise it any higher and don't think we should need to -- we
want to find the root cause of this problem, since it hasn't happened before.

Can anyone tell me:
1. Do applications share the JVM? If so, can we turn that off?
2. How best to isolate the root cause of the problem? By the time we see the
error in the web server logs, ps -efl shows no processes eating a ton of
memory and the app has already fixed itself.

Any insights appreciated.

--
--
~kaeli~
Support your local medical examiner: die strangely!
http://www.ipwebdesign.net/wildAtHeart
http://www.ipwebdesign.net/kaelisSpace

 
Reply With Quote
 
 
 
 
Wibble
Guest
Posts: n/a
 
      04-25-2005
Run your program in the debugger (or attach to it) and trap on
java.lang.OutOfMemory.

kaeli wrote:
> Very recently, a web application ( JSP / Iplanet6 / Solaris 2. has begun to
> freeze up. Web server logs indicate:
> Internal error: Unexpected error condition thrown
> java.lang.OutOfMemoryError,no description), stack: java.lang.OutOfMemoryError
>
> This happens intermittently, with no obvious cause, and it just started
> happening this morning (no changes have been made to either the server or the
> application). It resolves itself and the application resumes just fine. But
> it's irritating for pages to freeze up on people.
>
> I *suspect* another application is using too much memory, but can't figure
> out if that is the case.
> Raising the JVM heap (in server admin GUI) to 256 didn't help. We can't /
> don't want to raise it any higher and don't think we should need to -- we
> want to find the root cause of this problem, since it hasn't happened before.
>
> Can anyone tell me:
> 1. Do applications share the JVM? If so, can we turn that off?
> 2. How best to isolate the root cause of the problem? By the time we see the
> error in the web server logs, ps -efl shows no processes eating a ton of
> memory and the app has already fixed itself.
>
> Any insights appreciated.
>

 
Reply With Quote
 
 
 
 
Robert
Guest
Posts: n/a
 
      04-26-2005
You should also try JProbe that helped us out here tremendously. And
no apps don't share the JVM.

 
Reply With Quote
 
kaeli
Guest
Posts: n/a
 
      04-26-2005
In article <(E-Mail Removed) .com>,
http://www.velocityreviews.com/forums/(E-Mail Removed) enlightened us with...
> You should also try JProbe that helped us out here tremendously. And
> no apps don't share the JVM.
>


The problem mysteriously went away into the void from which it came.
It definitely looks like the problem was caused by another application. If
they don't share the JVM, what other conditions might cause outOfMemoryError?
This is a medium server with 512 RAM and nearly unlimited virtual memory. The
machine admins were flummoxed, so little ole' me is trying to figure this
out.

--
--
~kaeli~
A man needs a mistress... just to break the monogamy.
http://www.ipwebdesign.net/wildAtHeart
http://www.ipwebdesign.net/kaelisSpace

 
Reply With Quote
 
kaeli
Guest
Posts: n/a
 
      04-26-2005
In article <(E-Mail Removed)>, (E-Mail Removed)
enlightened us with...
> Run your program in the debugger (or attach to it) and trap on
> java.lang.OutOfMemory.


How would I do that for JSP (IPlanet 6.0)?

And how would that tell me anything?
If I know exactly what was executing at the time the error occurred, it still
would only be the straw that broke the camel's back, not the actual cause of
the problem, quite likely. I saw the error occur at multiple times, on
different pages, at seemingly random times. The pages would then just work
again in a couple minutes. So I highly suspect that the real problem has
nothing to do with my application and everything to do with someone else's.
The error last occured yesterday at 10:24 am. It had occured on and off all
morning. But it's all gone now. Yet everyone is still using my application.

It's a mystery I'd like to solve just so I know WTF happened and can have it
not happen again.

--
--
~kaeli~
Suicide is the most sincere form of self-criticism.
http://www.ipwebdesign.net/wildAtHeart
http://www.ipwebdesign.net/kaelisSpace

 
Reply With Quote
 
Marcus Eaton
Guest
Posts: n/a
 
      04-26-2005
kaeli wrote:

>In article <(E-Mail Removed)>, (E-Mail Removed)
>enlightened us with...
>
>
>>Run your program in the debugger (or attach to it) and trap on
>>java.lang.OutOfMemory.
>>
>>

>
>How would I do that for JSP (IPlanet 6.0)?
>
>And how would that tell me anything?
>If I know exactly what was executing at the time the error occurred, it still
>would only be the straw that broke the camel's back, not the actual cause of
>the problem, quite likely. I saw the error occur at multiple times, on
>different pages, at seemingly random times. The pages would then just work
>again in a couple minutes. So I highly suspect that the real problem has
>nothing to do with my application and everything to do with someone else's.
>The error last occured yesterday at 10:24 am. It had occured on and off all
>morning. But it's all gone now. Yet everyone is still using my application.
>
>It's a mystery I'd like to solve just so I know WTF happened and can have it
>not happen again.
>
>
>


How? Use a java profiling tool, such as JProbe or JProfiler. I haven't
used JProbe, but with jprofiler you basically plug in remote debugging
capabilities into a java process, which you can then connect to via the
Jprofiler tool.

Profilers will be able to show you things like:
Where most memory is being used, which methods and classes etc
Ditto for time usage, and nifty things like that.

Once you have the profiler connected to your app server (A test server.
Probably not wise to do this on prod if you run a load testing tool,
or something, you can see where the most memory is being allocated, and
target that for investigation.

Not sure if there are any decent open-source / free java profiling
tools, anyone know of one?

Marcus
 
Reply With Quote
 
Gerbrand van Dieijen
Guest
Posts: n/a
 
      04-26-2005
kaeli schreef:

> The problem mysteriously went away into the void from which it came.
> It definitely looks like the problem was caused by another application. If
> they don't share the JVM, what other conditions might cause outOfMemoryError?
> This is a medium server with 512 RAM and nearly unlimited virtual memory. The
> machine admins were flummoxed, so little ole' me is trying to figure this
> out.


Java 1.4 and lower has an fixed maximum to the heap memory.
You can increase it by adding paramaters (see java -X for options and
the web). This won't explain the cause of the error, but might solve it.


 
Reply With Quote
 
kaeli
Guest
Posts: n/a
 
      04-26-2005
In article <X8rbe.27022$(E-Mail Removed)>, meaton1
@bigpond.net enlightened us with...
>
> Once you have the profiler connected to your app server (A test server.
> Probably not wise to do this on prod


See, that's the problem.
It isn't happening on the test server.

The problem has come back, with a vengence, and none of the web stuff is
functioning (even pure html pages and the web server admin GUI), so it's
looking more and more like a machine issue.

--
--
~kaeli~
Not one shred of evidence supports the notion that life is
serious.
http://www.ipwebdesign.net/wildAtHeart
http://www.ipwebdesign.net/kaelisSpace

 
Reply With Quote
 
kaeli
Guest
Posts: n/a
 
      04-26-2005
In article <8e920$426e5437$9161dfd0$(E-Mail Removed)> ,
(E-Mail Removed) enlightened us with...
> kaeli schreef:
>
> > The problem mysteriously went away into the void from which it came.
> > It definitely looks like the problem was caused by another application. If
> > they don't share the JVM, what other conditions might cause outOfMemoryError?
> > This is a medium server with 512 RAM and nearly unlimited virtual memory. The
> > machine admins were flummoxed, so little ole' me is trying to figure this
> > out.

>
> Java 1.4 and lower has an fixed maximum to the heap memory.
> You can increase it by adding paramaters (see java -X for options and
> the web). This won't explain the cause of the error, but might solve it.


We tried that already. First thing I tried, actually.

I posted in another reply that now the machine isn't even rendering static
html pages properly, so it's looking more like a machine issue than a java
issue. Everything is locked up at the moment.
I was just the first to notice the budding problems, apparently. Lucky me.


--
--
~kaeli~
"When dogma enters the brain, all intellectual activity
ceases" -- Robert Anton Wilson
http://www.ipwebdesign.net/wildAtHeart
http://www.ipwebdesign.net/kaelisSpace

 
Reply With Quote
 
John C. Bollinger
Guest
Posts: n/a
 
      04-26-2005
kaeli wrote:

> In article <(E-Mail Removed) .com>,
> (E-Mail Removed) enlightened us with...
>
>>You should also try JProbe that helped us out here tremendously. And
>>no apps don't share the JVM.
>>

>
>
> The problem mysteriously went away into the void from which it came.
> It definitely looks like the problem was caused by another application. If
> they don't share the JVM, what other conditions might cause outOfMemoryError?
> This is a medium server with 512 RAM and nearly unlimited virtual memory. The
> machine admins were flummoxed, so little ole' me is trying to figure this
> out.


Java applications do not share a VM unless you do work to cause it to be
so. There is, however, some question as to how to define "application."
In this case the "application" is your IPlanet application server,
which conceivably may host more than one service (avoiding the term
"application" for this). All services hosted by IPlanet share a VM (or
multiple VMs in a clustering scenario, but you indicated that there was
only one server). Even if you have only installed one application of
your own on the server, it may come with built-in services (such as
management services) that consume VM resources when active.

The amount of RAM and virtual memory on the box is irrelevant, so long
as it is sufficient to support the Java VM's maximum heap, with enough
to spare to keep the rest of the box running. The machine will throw
OOME if the VM is unable to satisfy an allocation request despite its
best effort to free heap space by expanding the heap (if possible) and
performing garbage collection, and the only way this happens is if the
application server and the services running in it fill up the heap with
reachable objects. It may be possible to recover from the OOME if it
causes enough stack frames to pop and / or enough threads to die that
there is once again sufficient free heap.

There are a few reasonably likely scenarios here:

(1) Your application has an acute packratting problem (i.e. it holds on
to large objects past its need for them, but the problem is localized).
There are some gatchas related to String handling that are a
reasonably good generic candidate here if your application ever deals
with large Strings. A problem such as this could be data-dependent,
which could explain why it didn't manifest before and now has stopped
manifesting -- it might have been related to what some particular client
was doing with the app.

(2) Your application has a chronic packratting problem (i.e. it holds on
to medium sized objects long past its need for them -- possibly
forever). Much of the above applies here as well, but the problem might
not have strong dependencies on specific data. It just slowly fills up
the heap until there is nothing left. I'm assuming here that the OOME
might kill the whole service, but that IPlanet recognizes that and
restarts it.

(3) Your server may have been subject to a DOS attack. A reasonable
level of logging ought to show this up, though it's possible that the
memory situation could make logging fail.


Overall, I have one general piece of advice: when debugging a service,
look first at the logs. You've already done some of that, but you may
not have considered _all_ the logs. For instance, does IPlanet keep
access logs separate from service and / or error logs? The access logs
are a key part of the picture: what was being requested of the service
when it failed?

I hope some of that helps.

--
John Bollinger
(E-Mail Removed)

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Interesting segmentation fault, I cannot find root cause b3hzat C Programming 2 07-21-2010 12:20 AM
Look2Me root cause? ng_reader Computer Support 10 01-24-2006 04:08 AM
will all these messages cause a problem . I am a new subscriber and my computer is downloading 100,000 messages. Will this cause any kind of a problem with my ability to store other items?? Camille White Camille White Computer Support 9 11-08-2004 01:13 AM
Root cause of ASP .NET restarting... Scott Lezberg ASP .Net 1 08-12-2004 03:27 AM
A possible cause of this error is insufficient access rights to perform the operation. You can manually configure the application root ... Arturo ASP .Net Security 0 02-06-2004 09:55 AM



Advertisments