Symptom:
Web team reports that request to tomcat is not responding all of a sudden. The environment is an isolated one with access control, which means there is no system or application change recently.
Application Log:
Caused by: com.inet.ora.Ora4SQLException: [Oracle] #1 ORA-00257: archiver error. Connect internal only, until freed.
[Oracle] #1
at com.inet.ora.i.a(Unknown Source)
at com.inet.ora.OraConnection.a(Unknown Source)
at com.inet.ora.OraConnection.q(Unknown Source)
at com.inet.ora.OraConnection.<init>(Unknown Source)
at com.inet.ora.l.<init>(Unknown Source)
at com.inet.ora.i.a(Unknown Source)
at com.inet.ora.OraDriver.connect(Unknown Source)
at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:294)
at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1247)
at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1221)
... 46 more
Thread Dump:
"http-8080-12" daemon prio=10 tid=0x0000000046b20c00 nid=0x18e0 runnable [0x000000004954e000..0x000000004954fb90]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.inet.ora.OraConnection.i(Unknown Source)
at com.inet.ora.OraConnection.a(Unknown Source)
- locked <0x00002aab89aad9d0> (a com.inet.ora.l)
at com.inet.ora.ax.a(Unknown Source)
at com.inet.ora.ax.c(Unknown Source)
at com.inet.ora.ax.execute(Unknown Source)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeUpdate(SqlExecutor.java:81)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteUpdate(GeneralStatement.java:200)
at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeUpdate(GeneralStatement.java:78)
at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.update(SqlMapExecutorDelegate.java:505)
at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.update(SqlMapSessionImpl.java:90)
at com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.update(SqlMapClientImpl.java:67)
Root Cause:
Oracle archived redo logs used up disk space. Linux 'df' shows the space is 100% used.
Solution:
As the log mentioned, freed space. We did a quick move of archived log files to another disk for the fix. Meanwhile, we are discussing 2 options: either turn on NOARCHIVELOG, or write a script to do automatic gzip and move to other disk.
Thanks for the post.
ReplyDeleteJava training in Pune