Monday, April 30, 2012

Apache DBCP deadlock

commons-dbcp-1.2.2 (http://commons.apache.org/dbcp/api-1.2.2/index.html) has a bug https://issues.apache.org/jira/browse/DBCP-281 which can cause deadlock when Evictor thread validating connection while another thread is closing the same connection. When the deadlock happens, all threads using DB connections will hang unless a restart of application server.

"Timer-0":
        at org.apache.commons.dbcp.AbandonedTrace.addTrace(AbandonedTrace.java:175)
        - waiting to lock <0x00002aab1c09e780> (a org.apache.commons.dbcp.PoolableConnection)
        at org.apache.commons.dbcp.AbandonedTrace.init(AbandonedTrace.java:92)
        at org.apache.commons.dbcp.AbandonedTrace.(AbandonedTrace.java:82)
        at org.apache.commons.dbcp.DelegatingStatement.(DelegatingStatement.java:61)
        at org.apache.commons.dbcp.DelegatingConnection.createStatement(DelegatingConnection.java:224)
        at org.apache.commons.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:331)
        at org.apache.commons.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312)
        at org.apache.commons.pool.impl.GenericObjectPool.evict(GenericObjectPool.java:1217)
        - locked <0x00002aaae61d4038> (a org.apache.commons.pool.impl.GenericObjectPool)
        at org.apache.commons.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1341)
        at java.util.TimerThread.mainLoop(Timer.java:512)
        at java.util.TimerThread.run(Timer.java:462)


"http-8083-11":
        at org.apache.commons.pool.impl.GenericObjectPool.addObjectToPool(GenericObjectPool.java:1137)
        - waiting to lock <0x00002aaae61d4038> (a org.apache.commons.pool.impl.GenericObjectPool)
        at org.apache.commons.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java:1076)
        at org.apache.commons.dbcp.PoolableConnection.close(PoolableConnection.java:87)
        - locked <0x00002aab1c09e780> (a org.apache.commons.dbcp.PoolableConnection)
        at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:181)



"http-8083-150":
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:916)
        - waiting to lock <0x00002aaae61d4038> (a org.apache.commons.pool.impl.GenericObjectPool)
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96)
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880)
Two solutions to this problem:
1. Remove testWhileIdle="true", use default instead.
2. Upgrade commons-dbcp library to latest version, or use other libraries like c3p0, proxool, BoneCP etc.

No comments:

Post a Comment