Connection pooling problems

Since moving to using Tomcat from Resin for our production server and using spring templates for database connections the database connection pooling started to suffer from stale connections resulting in errors like this:

 

java.sql.SQLException: Communication link failure: java.io.EOFException, underlying cause: null

** BEGIN NESTED EXCEPTION **

java.io.EOFException

STACKTRACE:

java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1395)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:1539)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:1930)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1168)

 

After hunting around for a solution I found that adding the following parameters to the database configuration in JNDI in context.xml solved the issue which would eventually cause a permgen out of memory error completely freezing tomcat and the VM it was running on:

validationQuery=”select 1″ testOnBorrow=”true”

 

copied below is the post I found that was most useful in solving this from http://stackoverflow.com/questions/1448974/problem-with-connection-pooling-with-java-and-mysql-in-tomcat-web-application:

 

There are a few pointers on avoiding this situation, obtained from other sources, especially from the connection pool implementations of other drivers and from other application servers. Some of the information is already available in the Tomcat documentation on JNDI Data Sources.

  1. Establish a cleanup/reaper schedule that will close connections in the pool, if they are inactive beyond a certain period. It is not good practice to leave a connection to the database open for 8 hours (the MySQL default). On most application servers, the inactive connection timeout value is configurable and is usually less than 15 minutes (i.e. connections cannot be left in the pool for more than 15 minutes unless they are being reused time and again). In Tomcat, when using a JNDI DataSource, use the removeAbandoned and removeAbandonedTimeout settings to do the same.
  2. When a new connection is return from the pool to the application, ensure that it is tested first. For instance, most application servers that I know, can be configured so that connection to an Oracle database are tested with an execute of “SELECT 1 FROM dual”. In Tomcat, use the validationQuery property to set the appropriate query for MySQL – I believe this is “SELECT 1” (without quotes). The reason why setting the value of the validationQuery property helps, is because if the query fails to execute, the connection is dropped from the pool, and new one is created in its place.

As far are the behavior of your application is concerned, the user is probably seeing the result of the pool returning a stale connection to the application for the first time. The second time around, the pool probably returns a different connection that can service the application’s queries.

Tomcat JNDI Data Sources are based on Commons DBCP, so the configuration properties applicable to DBCP will apply to Tomcat as well.

Advertisements
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: