«

Hi,
We are running a web application which is a stateless system and
gets a lot of hits a day per server.
We are running the following configuration :
Red Hat Linux 7.3
Linux Kernel 2.4.20-19.7smp
Sun JVM JDK 1.4.1_02
The machine has 2GB of RAM and we have set the -XMx=1500M.

Pretty much everyday after about 200,000 - 250,000 hits, the
server get the error shown below. We have looked at the memory usage
and it looks ok.The application does load up some data and use a
substantial amount of memory(about 500M). But we looking through both
top and Runtime.getRuntime().totalMemory() we see that it has still
not close to the 1500M limit that we have set. Also we use the
following code on a jsp page to montior the threads and they don't
seem to be spiralling out of control

Thread[] threads = new Thread[1000];
ThreadGroup tg = Thread.currentThread().getThreadGroup();

while (tg.getParent() != null)
{
tg = tg.getParent();
}

tg.enumerate( threads );
for (int i = 0; i < threads.length; i++)
{
if (threads[i] != null)
{
out.print(i + ": ");
out.print( "pri=" + threads[i].getPriority() + " ");
out.print( "ac=" + threads[i].activeCount() + " ");
out.println(threads[i]);
}
}

Does anyone have any idea what the problem could be ? What the
possible issues could be ? We have looked at Memory and threads and
they don't seem to be the issue. Is here any documentation about the
cases that would through up this error ? Any help in this matter would
be greatly appreciated.

Thanks,
Mit








javax.servlet.ServletException: Servlet execution threw an exception
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(Unknown
Source)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(Unknown
Source)
at org.apache.catalina.core.StandardWrapperValve.invo ke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
at org.apache.catalina.core.StandardContextValve.invo ke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
at org.apache.catalina.core.StandardContext.invoke(Un known Source)
at org.apache.catalina.core.StandardHostValve.invoke( Unknown Source)
at org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.valves.ErrorDispatcherValve.in voke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.valves.ErrorReportValve.invoke (Unknown Source)
at org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
at org.apache.catalina.core.StandardEngineValve.invok e(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(Co yoteAdapter.java:223)
at org.apache.coyote.http11.Http11Processor.process(H ttp11Processor.java:594)
at org.apache.coyote.http11.Http11Protocol$Http11Conn ectionHandler.processConnection(Http11Protocol.jav a:392)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(P oolTcpEndpoint.java:565)
at org.apache.tomcat.util.threads.ThreadPool$ControlR unnable.run(ThreadPool.java:619)
at java.lang.Thread.run(Thread.java:536)
----- Root Cause -----
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start(Native Method)
at com.i411.search.controller.LowLevelSearch.performS ocketSearch(LowLevelSearch.java:42
at com.i411.search.controller.LowLevelSearch.performN etworkSearch(LowLevelSearch.java:34
at com.i411.search.controller.LowLevelSearch.ExecuteS earch(LowLevelSearch.java:145)
at com.i411.search.controller.SearchController.catIte msSearch(SearchController.java:407)
at com.i411.search.controller.SearchController.getSea rchResults(SearchController.java:84)
at com.i411.dex.searchflow.SearchExecutor.run(SearchE xecutor.java:45)
at com.i411.dex.searchflow.BusinessNameSearchExecutor .run(BusinessNameSearchExecutor.java:82)
at com.i411.dex.searchflow.DexSearchFlow.performSearc hFlow(DexSearchFlow.java:335)
at com.i411.web.common.WebController.executeSearchWit hSearchFlow(WebController.java:137)
at com.i411.web.common.WebController.executeSearchWit hSearchFlow(WebController.java:182)
at com.i411.dex.xml.XMLSearchServlet.PerformQuerySear ch(XMLSearchServlet.java:260)
at com.i411.dex.xml.XMLSearchServlet.doGet(XMLSearchS ervlet.java:17
at com.i411.dex.xml.XMLSearchServlet.doPost(XMLSearch Servlet.java:366)
at javax.servlet.http.HttpServlet.service(HttpServlet .java)
at javax.servlet.http.HttpServlet.service(HttpServlet .java)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(Unknown
Source)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(Unknown
Source)
at org.apache.catalina.core.StandardWrapperValve.invo ke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
at org.apache.catalina.core.StandardContextValve.invo ke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
at org.apache.catalina.core.StandardContext.invoke(Un known Source)
at org.apache.catalina.core.StandardHostValve.invoke( Unknown Source)
at org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.valves.ErrorDispatcherValve.in voke(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.valves.ErrorReportValve.invoke (Unknown Source)
at org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
at org.apache.catalina.core.StandardEngineValve.invok e(Unknown
Source)
at org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invokeNext(Unknown
Source)
at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(Co yoteAdapter.java:223)
at org.apache.coyote.http11.Http11Processor.process(H ttp11Processor.java:594)
at org.apache.coyote.http11.Http11Protocol$Http11Conn ectionHandler.processConnection(Http11Protocol.jav a:392)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(P oolTcpEndpoint.java:565)
at org.apache.tomcat.util.threads.ThreadPool$ControlR unnable.run(ThreadPool.java:619)
at java.lang.Thread.run(Thread.java:536)

Hi !

We also encountered this problem before.

You will probably have to play with the "ulimit" command.

Regards,

Arnaud

"Mit Majithia" <> a écrit dans le message news:
...
> Hi,
> We are running a web application which is a stateless system and
> gets a lot of hits a day per server.
> We are running the following configuration :
> Red Hat Linux 7.3
> Linux Kernel 2.4.20-19.7smp
> Sun JVM JDK 1.4.1_02
> The machine has 2GB of RAM and we have set the -XMx=1500M.
>
> Pretty much everyday after about 200,000 - 250,000 hits, the
> server get the error shown below. We have looked at the memory usage
> and it looks ok.The application does load up some data and use a
> substantial amount of memory(about 500M). But we looking through both
> top and Runtime.getRuntime().totalMemory() we see that it has still
> not close to the 1500M limit that we have set. Also we use the
> following code on a jsp page to montior the threads and they don't
> seem to be spiralling out of control
>
> Thread[] threads = new Thread[1000];
> ThreadGroup tg = Thread.currentThread().getThreadGroup();
>
> while (tg.getParent() != null)
> {
> tg = tg.getParent();
> }
>
> tg.enumerate( threads );
> for (int i = 0; i < threads.length; i++)
> {
> if (threads[i] != null)
> {
> out.print(i + ": ");
> out.print( "pri=" + threads[i].getPriority() + " ");
> out.print( "ac=" + threads[i].activeCount() + " ");
> out.println(threads[i]);
> }
> }
>
> Does anyone have any idea what the problem could be ? What the
> possible issues could be ? We have looked at Memory and threads and
> they don't seem to be the issue. Is here any documentation about the
> cases that would through up this error ? Any help in this matter would
> be greatly appreciated.
>
> Thanks,
> Mit
>
>
>
>
>
>
>
>
> javax.servlet.ServletException: Servlet execution threw an exception
> at
org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(Unknown
> Source)
> at org.apache.catalina.core.ApplicationFilterChain.do Filter(Unknown
> Source)
> at org.apache.catalina.core.StandardWrapperValve.invo ke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
> at org.apache.catalina.core.StandardContextValve.invo ke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
> at org.apache.catalina.core.StandardContext.invoke(Un known Source)
> at org.apache.catalina.core.StandardHostValve.invoke( Unknown Source)
> at
org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.valves.ErrorDispatcherValve.in voke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.valves.ErrorReportValve.invoke (Unknown Source)
> at
org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
> at org.apache.catalina.core.StandardEngineValve.invok e(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
> at org.apache.coyote.tomcat4.CoyoteAdapter.service(Co yoteAdapter.java:223)
> at
org.apache.coyote.http11.Http11Processor.process(H ttp11Processor.java:594)
> at
org.apache.coyote.http11.Http11Protocol$Http11Conn ectionHandler.processConne
ction(Http11Protocol.java:392)
> at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(P oolTcpEndpoint.java:565)
> at
org.apache.tomcat.util.threads.ThreadPool$ControlR unnable.run(ThreadPool.jav
a:619)
> at java.lang.Thread.run(Thread.java:536)
> ----- Root Cause -----
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start(Native Method)
> at
com.i411.search.controller.LowLevelSearch.performS ocketSearch(LowLevelSearch
..java:42
> at
com.i411.search.controller.LowLevelSearch.performN etworkSearch(LowLevelSearc
h.java:34
> at
com.i411.search.controller.LowLevelSearch.ExecuteS earch(LowLevelSearch.java:
145)
> at
com.i411.search.controller.SearchController.catIte msSearch(SearchController.
java:407)
> at
com.i411.search.controller.SearchController.getSea rchResults(SearchControlle
r.java:84)
> at com.i411.dex.searchflow.SearchExecutor.run(SearchE xecutor.java:45)
> at
com.i411.dex.searchflow.BusinessNameSearchExecutor .run(BusinessNameSearchExe
cutor.java:82)
> at
com.i411.dex.searchflow.DexSearchFlow.performSearc hFlow(DexSearchFlow.java:3
35)
> at
com.i411.web.common.WebController.executeSearchWit hSearchFlow(WebController.
java:137)
> at
com.i411.web.common.WebController.executeSearchWit hSearchFlow(WebController.
java:182)
> at
com.i411.dex.xml.XMLSearchServlet.PerformQuerySear ch(XMLSearchServlet.java:2
60)
> at com.i411.dex.xml.XMLSearchServlet.doGet(XMLSearchS ervlet.java:17
> at com.i411.dex.xml.XMLSearchServlet.doPost(XMLSearch Servlet.java:366)
> at javax.servlet.http.HttpServlet.service(HttpServlet .java)
> at javax.servlet.http.HttpServlet.service(HttpServlet .java)
> at
org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(Unknown
> Source)
> at org.apache.catalina.core.ApplicationFilterChain.do Filter(Unknown
> Source)
> at org.apache.catalina.core.StandardWrapperValve.invo ke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
> at org.apache.catalina.core.StandardContextValve.invo ke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
> at org.apache.catalina.core.StandardContext.invoke(Un known Source)
> at org.apache.catalina.core.StandardHostValve.invoke( Unknown Source)
> at
org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.valves.ErrorDispatcherValve.in voke(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.valves.ErrorReportValve.invoke (Unknown Source)
> at
org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
> at org.apache.catalina.core.StandardEngineValve.invok e(Unknown
> Source)
> at
org.apache.catalina.core.StandardPipeline$Standard PipelineValveContext.invok
eNext(Unknown
> Source)
> at org.apache.catalina.core.StandardPipeline.invoke(U nknown Source)
> at org.apache.catalina.core.ContainerBase.invoke(Unkn own Source)
> at org.apache.coyote.tomcat4.CoyoteAdapter.service(Co yoteAdapter.java:223)
> at
org.apache.coyote.http11.Http11Processor.process(H ttp11Processor.java:594)
> at
org.apache.coyote.http11.Http11Protocol$Http11Conn ectionHandler.processConne
ction(Http11Protocol.java:392)
> at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(P oolTcpEndpoint.java:565)
> at
org.apache.tomcat.util.threads.ThreadPool$ControlR unnable.run(ThreadPool.jav
a:619)
> at java.lang.Thread.run(Thread.java:536)

Have encouneterd a similar problem on HP-UX 11. Check for values of
the soft and hard limits (kernel settings) on the number of threads
allowed per process on your system. These limits may have been set to
a low value on your system.
Reagrds,
Jay

Mit Majithia wrote:

> Hi,
> We are running a web application which is a stateless system and
> gets a lot of hits a day per server.
> We are running the following configuration :
> Red Hat Linux 7.3
> Linux Kernel 2.4.20-19.7smp
> Sun JVM JDK 1.4.1_02
> The machine has 2GB of RAM and we have set the -XMx=1500M.

The higher you set -mx, the less threads you can create (why does a
stateless system need 1.5G anyway ...)

I think a very naive formula is max threads = (2G - mx) / stacksize.
You may try to work with a smaller heap size or smaller stack sizes.

Matthias
--
Matthias Ernst
CoreMedia - The Content Technology Experts

"Toll takes its time."

Update:
We have been playing around with the settings.One of the
settings that we changed was to reduce the memory allocated to the JVM
(-Xmx1500m) to be lesser to -Xmx1000m. This seems to have alleviated
the situation a little for the time being. It does not go down and
does not give the error as before. Also we do have to restart the
server on a daily basis to update the data.But with the new settings
it makes it easily through the entire day without giving an error. Can
anyone explain this ? Due to the growing traffic and data we will
probably need the expanded memory sometime in the future. We feel that
at the moment we have just pushed the problem to surface at some point
in the future.

Thanks,
Mit

"Mit Majithia" <> wrote in message
news: om...
> Update:
> We have been playing around with the settings.One of the
> settings that we changed was to reduce the memory allocated to the JVM
> (-Xmx1500m) to be lesser to -Xmx1000m. This seems to have alleviated
> the situation a little for the time being. It does not go down and
> does not give the error as before. Also we do have to restart the
> server on a daily basis to update the data.But with the new settings
> it makes it easily through the entire day without giving an error. Can
> anyone explain this ? Due to the growing traffic and data we will
> probably need the expanded memory sometime in the future. We feel that
> at the moment we have just pushed the problem to surface at some point
> in the future.
>
> Thanks,
> Mit

The Java process has two memory areas: the Java heap, and the "native heap",
which combine total the memory usage of the process.
The Java heap is controlled via the -Xms and -Xmx setting, and the space
available to the native heap is that which isn't used by the Java heap.

The act of reducing the maximum Java heap size has made the "native heap"
bigger, and this is the area that was memory constrained.
We know this because the OutOfMemoryError was generated the message informed
you that the JVM was unable to allocate a new native stack, this is
allocated onto the native heap (there is also a Java thread object which is
created and allocated onto the Java heap).

It is entirely possible that the amount of "native heap" available to the
JVM was insufficient to allocate the underlying resources to run the Java
process under the load that was being driven through it. The native heap is
now 500MB bigger, and unless there is a memory leak or the load is
significantly increased, this change should prevent any OutOfMemoryErrors
based on the native heap.

This could be a couple of different things.

What are the full JVM options you are running with? Do you have -Xms
specified also, and are you using any custom PermGen settings? You may
want to try setting -Xms to the same value as -Xmx.

Try using JVMSTAT on the process and check to see what is actually
happening in the VM (as per young generation, old generation, etc.).
JVMSTAT will help you visualize what is happening with memory inside the
VM. This will help you figure out what you need to tune.

To get jvmstat, http://developers.sun.com/dev/coolstuff/jvmstat/ .

Otherwise, if you just want to manually look at the garbage collection
information you can add flags such as '-verbose:gc' and
'XX:+PrintGCDetails'. You can also output the GC information to a file
using '-Xloggc:<file>'.

-Moazam

Mit Majithia wrote:

> Hi,
> We are running a web application which is a stateless system and
> gets a lot of hits a day per server.
> We are running the following configuration :
> Red Hat Linux 7.3
> Linux Kernel 2.4.20-19.7smp
> Sun JVM JDK 1.4.1_02
> The machine has 2GB of RAM and we have set the -XMx=1500M.
>
> Pretty much everyday after about 200,000 - 250,000 hits, the
> server get the error shown below. We have looked at the memory usage
> and it looks ok.The application does load up some data and use a
> substantial amount of memory(about 500M). But we looking through both
> top and Runtime.getRuntime().totalMemory() we see that it has still
> not close to the 1500M limit that we have set. Also we use the
> following code on a jsp page to montior the threads and they don't
> seem to be spiralling out of control
>
> Thread[] threads = new Thread[1000];
> ThreadGroup tg = Thread.currentThread().getThreadGroup();
>
> while (tg.getParent() != null)
> {
> tg = tg.getParent();
> }
>
> tg.enumerate( threads );
> for (int i = 0; i < threads.length; i++)
> {
> if (threads[i] != null)
> {
> out.print(i + ": ");
> out.print( "pri=" + threads[i].getPriority() + " ");
> out.print( "ac=" + threads[i].activeCount() + " ");
> out.println(threads[i]);
> }
> }
>
> Does anyone have any idea what the problem could be ? What the
> possible issues could be ? We have looked at Memory and threads and
> they don't seem to be the issue. Is here any documentation about the
> cases that would through up this error ? Any help in this matter would
> be greatly appreciated.
>
> Thanks,
> Mit
>
>
>
>
>
>
>
>
> ----- Root Cause -----
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start(Native Method)
> at com.i411.search.controller.LowLevelSearch.performS ocketSearch(LowLevelSearch.java:42
> at com.i411.search.controller.LowLevelSearch.performN etworkSearch(LowLevelSearch.java:34
...............