Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Thread Dump analysis | ojdbc14 | Oracle Connection pool

Reply
Thread Tools

Thread Dump analysis | ojdbc14 | Oracle Connection pool

 
 
kiranmalla.in@gmail.com
Guest
Posts: n/a
 
      11-23-2005
Hi

The problem description is a bit large...wanted to be more descriptive.

System Architecture:
-------------------
Apache, Tomcats 5.0.x, mod-jk (for load balancing), Oracle, jsp, java,
ojdbc14 for connection pooling, jdk.14.1

Issue:
------
We are running a cluster of 8 tomcats and 4 apache instances load
balanced by mod-jk. Some Tomcats stop responding after running for a
period of time and need a restart. We have observed over a period of
time that a restart of tomcat is required when the thread count is
around 543 and heap is maxed out at 990M. We are giving application
intial memory of 1 GB using the Xms and Xmx settings while starting
tomcats...so memory doesn't seem to be an issue.

Analysis:
---------
We have started recording the thread dumps. Most of the thread dumps
show a similar behaviour. Let's say 500 threads "waiting for a monitor
which is locked by a thread

The messages in 500 threads is "- waiting to lock <0xbf4e9e60> (a
oracle.jdbc.pool.OracleConnectionEventListener)"

This thread has the lock:
"TP-Processor633" daemon prio=5 tid=0x004921d0 nid=0x5ba waiting for
monitor entry [872fe000..872ffc28]
at
oracle.jdbc.pool.OracleConnectionCacheImpl.closeSi ngleConnection(OracleConnectionCacheImpl.java:677)
- waiting to lock <0xbf4e98f8> (a
oracle.jdbc.pool.OracleConnectionCacheImpl)
at
oracle.jdbc.pool.OracleConnectionCacheImpl.reusePo oledConnection(OracleConnectionCacheImpl.java:592)
at
oracle.jdbc.pool.OracleConnectionEventListener.con nectionClosed(OracleConnectionEventListener.java:1 30)
- locked <0xbf4e9e60> (a
oracle.jdbc.pool.OracleConnectionEventListener)
at
oracle.jdbc.pool.OraclePooledConnection.callListen er(OraclePooledConnection.java:482)
at
oracle.jdbc.pool.OraclePooledConnection.logicalClo se(OraclePooledConnection.java:445)
- locked <0xcc9af820> (a oracle.jdbc.pool.OraclePooledConnection)
at
oracle.jdbc.driver.OracleConnection.logicalClose(O racleConnection.java:2936)
- locked <0xcc9af8c8> (a oracle.jdbc.driver.OracleConnection)
at
oracle.jdbc.driver.OracleConnection.close(OracleCo nnection.java:1454)
- locked <0xcc9af8c8> (a oracle.jdbc.driver.OracleConnection)
at com.ss.framework.sql.SqlStatement.close(SqlStateme nt.java:984)
at
com.cc.ac.cnams.classmanagement.ClassQry.getCommun ityLinksForCourse(ClassQry.java:15796)
at
com.cc.ac.cnams.classmanagement.ClassQry.getClassD etailsForViewClassInfo(ClassQry.java:8069)

Threads waiting for the above lock to be released (around 500 in
number)
"TP-Processor632" daemon prio=5 tid=0x00491578 nid=0x5b9 waiting for
monitor entry [873fd000..873ffc28]
at
oracle.jdbc.pool.OracleConnectionEventListener.con nectionClosed(OracleConnectionEventListener.java:1 29)
- waiting to lock <0xbf4e9e60> (a
oracle.jdbc.pool.OracleConnectionEventListener)
at
oracle.jdbc.pool.OraclePooledConnection.callListen er(OraclePooledConnection.java:482)
at
oracle.jdbc.pool.OraclePooledConnection.logicalClo se(OraclePooledConnection.java:445)
- locked <0xcd699580> (a oracle.jdbc.pool.OraclePooledConnection)
at
oracle.jdbc.driver.OracleConnection.logicalClose(O racleConnection.java:2936)
- locked <0xcd699640> (a oracle.jdbc.driver.OracleConnection)
at
oracle.jdbc.driver.OracleConnection.close(OracleCo nnection.java:1454)
- locked <0xcd699640> (a oracle.jdbc.driver.OracleConnection)
at com.ss.framework.sql.SqlStatement.close(SqlStateme nt.java:984)
at
com.cc.ac.cnams.pa.PersonalAssociationQry.getRole( PersonalAssociationQry.java:1141)
at
com.cc.ac.cnams.pa.PersonalAssociation.getRole(Per sonalAssociation.java:2353)
at
com.cc.ac.cnams.entitlements.Entitlements.isUserOf Role(Entitlements.java:195)
at
org.apache.jsp.content.BannerLeftNav_jsp._jspServi ce(BannerLeftNav_jsp.java:201)
at com.ss.framework.presentation.servlet.Jsp.process( Jsp.java:49)
at com.ss.framework.presentation.servlet.Jsp.service( Jsp.java:41)
at
org.apache.jasper.servlet.JspServletWrapper.servic e(JspServletWrapper.java:324)
at
org.apache.jasper.servlet.JspServlet.serviceJspFil e(JspServlet.java:292)


I can provide the complete thread trace if anyone is interested in the
complete dump.

In the application, we are using ojdbc14 (oracle's connection pool) for
handling the connections. I have observed that after doing all the
operations in a DB object, we close the prepared statement
(SqlPreparedStatement.close). RESULT SET is not closed explicitly. Can
this issue be due to the resultset not being closed explicitly.

Javadoc for jdk 1.4.2 says: "A Statement object is automatically
closed when it is garbage collected. When a Statement object is closed,
its current ResultSet object, if one exists, is also closed.

"http://java.sun.com/j2se/1.4.2/docs/api/java/sql/Statement.html#close()


However, some doubts have been raised by people in the links below:
http://www.theserverside.com/discuss...hread_id=33592
http://forum.java.sun.com/thread.jsp...sageID=2615181

I'm not sure if not closing the 'REsult Set' explicitly can create this
type of issue as it is supposed to be handled by the java. Not sure if
ojdbc14 has implemented it as per the spec though.

Any pointers around the thread dump or the issue of whether ResultSet
should be closed explicity while using ORACLE Conneciton pool (ojdbc14)
would be higly appreciated. I can provide more details if required.

Thanks
Kiran

 
Reply With Quote
 
 
 
 
Bjorn Abelli
Guest
Posts: n/a
 
      11-23-2005

<(E-Mail Removed)> wrote...

> In the application, we are using ojdbc14 (oracle's connection pool) for
> handling the connections. I have observed that after doing all the
> operations in a DB object, we close the prepared statement
> (SqlPreparedStatement.close). RESULT SET is not closed explicitly. Can
> this issue be due to the resultset not being closed explicitly.


[snip]

> Any pointers around the thread dump or the issue of whether ResultSet
> should be closed explicity while using ORACLE Conneciton pool (ojdbc14)
> would be higly appreciated. I can provide more details if required.


Oracle claims that there are problems with some JVMs on the issue of
finalizers, therefore they urge users to explicitly close ResultSets.

// Bjorn A



 
Reply With Quote
 
 
 
 
kiranmalla.in@gmail.com
Guest
Posts: n/a
 
      11-23-2005
Thanks Bjorn,

Some additional info:
We are using sun'
"java version "1.4.1_02a"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1_02a-b01)
Java HotSpot(TM) Client VM (build 1.4.1_02a-b01, mixed mode)"

Is the Oracle recommendation documented at some place, so that I can
refer to that and make a case for closing the resultSets explicitly?

A strange behaviour in most of the threads (of the thread dump) is that
they have kept a lock at "
oracle.jdbc.pool.OracleConnectionCacheImpl.closeSi ngleConnection(OracleConnectionCacheImpl.java:677)
- waiting to lock <0xbf4e98f8> (a
oracle.jdbc.pool.OracleConnectionCacheImpl)"

I can understand if a thread is locked while getting a connection which
in turn can lead to other threads wait for the resource. But, I am
surprised as to why it is creating this issue, while closing the
connection.

Polling the group for any other inputs and known issues with the Oracle
connection pool.

Thanks
Kiran

 
Reply With Quote
 
Bjorn Abelli
Guest
Posts: n/a
 
      11-23-2005

<(E-Mail Removed)> wrote...

> Is the Oracle recommendation documented at some place,
> so that I can refer to that and make a case for closing
> the resultSets explicitly?


I would have given you a link to the documentation at Oracle, but it seems
like I would have to login to their technet-pages to find it, and I don't
have my password at hand, so I looked up a "mirror" instead...

http://techdoc.c-bizz.klopotek.de/cl...tml#close()%20

or...

http://tinyurl.com/9asac

....look what it says on the close() method...


// Bjorn A



 
Reply With Quote
 
Thomas Hawtin
Guest
Posts: n/a
 
      11-23-2005
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
>
> The messages in 500 threads is "- waiting to lock <0xbf4e9e60> (a
> oracle.jdbc.pool.OracleConnectionEventListener)"
>
> This thread has the lock:
> "TP-Processor633" daemon prio=5 tid=0x004921d0 nid=0x5ba waiting for
> monitor entry [872fe000..872ffc28]
> at
> oracle.jdbc.pool.OracleConnectionCacheImpl.closeSi ngleConnection(OracleConnectionCacheImpl.java:677)
> - waiting to lock <0xbf4e98f8> (a
> oracle.jdbc.pool.OracleConnectionCacheImpl)


Somewhere there may be a thread which holds that particular lock, which
may have a more interesting stack. You may have a situation where many
threads are acquiring and releasing the lock.

> In the application, we are using ojdbc14 (oracle's connection pool) for
> handling the connections. I have observed that after doing all the
> operations in a DB object, we close the prepared statement
> (SqlPreparedStatement.close). RESULT SET is not closed explicitly. Can
> this issue be due to the resultset not being closed explicitly.


The ResultSets should be closed at the latest when commit or rollback is
called (unless you use HOLD_SURSORS_OVER_COMMIT). However, in the past
some drivers have not been without bugs.

> Javadoc for jdk 1.4.2 says: "A Statement object is automatically
> closed when it is garbage collected. When a Statement object is closed,
> its current ResultSet object, if one exists, is also closed.


Don't rely on finalisation. Perhaps it will release the resource, but
you may already have run into trouble before it happens.

Tom Hawtin
--
Unemployed English Java programmer
http://jroller.com/page/tackline/
 
Reply With Quote
 
steve
Guest
Posts: n/a
 
      11-25-2005
On Wed, 23 Nov 2005 11:32:56 +0800, (E-Mail Removed) wrote
(in article <(E-Mail Removed). com>):

> Thanks Bjorn,
>
> Some additional info:
> We are using sun'
> "java version "1.4.1_02a"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.1_02a-b01)
> Java HotSpot(TM) Client VM (build 1.4.1_02a-b01, mixed mode)"
>
> Is the Oracle recommendation documented at some place, so that I can
> refer to that and make a case for closing the resultSets explicitly?
>
> A strange behaviour in most of the threads (of the thread dump) is that
> they have kept a lock at "
>

oracle.jdbc.pool.OracleConnectionCacheImpl.closeSi ngleConnection(OracleConnect

> ionCacheImpl.java:677)
> - waiting to lock <0xbf4e98f8> (a
> oracle.jdbc.pool.OracleConnectionCacheImpl)"
>
> I can understand if a thread is locked while getting a connection which
> in turn can lead to other threads wait for the resource. But, I am
> surprised as to why it is creating this issue, while closing the
> connection.
>
> Polling the group for any other inputs and known issues with the Oracle
> connection pool.
>
> Thanks
> Kiran
>


just a tip.

Explicitly close EVERYTHING you can.

Statments & result sets, and anything else you are using, ( apart from the
connection, if you are re-using it)

Basically if the oracle or SQL object has a ".close()" then use it
Explicitly.

Anything else is the kiss of death, but it looks like you may already know
that.

Steve



 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Thread control in thread pool for sending testisok Python 0 02-17-2009 07:34 AM
[Thread[pool-1-thread-2,5,main] - What does this mean? Hugo Java 4 03-27-2008 01:42 AM
install_driver(Oracle) failed: Can't load 'C:/Perl/site/lib/auto/DBD/Oracle/Oracle.dll' for module DBD::Oracle: load_file:The specified procedure could not be found at C:/Perl/lib/DynaLoader.pm line 230. Feyruz Perl Misc 4 10-14-2005 06:47 PM
source code for ojdbc14.jar Ulas Ergin Java 1 02-01-2005 02:10 PM
Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached. Guoqi Zheng ASP .Net 4 06-03-2004 06:39 PM



Advertisments