Friday, September 17, 2010

Tomcat is not responding due to ORA-00257

Symptom:
Web team reports that request to tomcat is not responding all of a sudden. The environment is an isolated one with access control, which means there is no system or application change recently.

Application Log:
Caused by: com.inet.ora.Ora4SQLException: [Oracle] #1 ORA-00257: archiver error. Connect internal only, until freed.
[Oracle] #1
    at com.inet.ora.i.a(Unknown Source)
    at com.inet.ora.OraConnection.a(Unknown Source)
    at com.inet.ora.OraConnection.q(Unknown Source)
    at com.inet.ora.OraConnection.<init>(Unknown Source)
    at com.inet.ora.l.<init>(Unknown Source)
    at com.inet.ora.i.a(Unknown Source)
    at com.inet.ora.OraDriver.connect(Unknown Source)
    at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
    at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:294)
    at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1247)
    at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1221)
    ... 46 more

Thread Dump:
"http-8080-12" daemon prio=10 tid=0x0000000046b20c00 nid=0x18e0 runnable [0x000000004954e000..0x000000004954fb90]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at com.inet.ora.OraConnection.i(Unknown Source)
    at com.inet.ora.OraConnection.a(Unknown Source)
    - locked <0x00002aab89aad9d0> (a com.inet.ora.l)
    at com.inet.ora.ax.a(Unknown Source)
    at com.inet.ora.ax.c(Unknown Source)
    at com.inet.ora.ax.execute(Unknown Source)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:169)
    at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeUpdate(SqlExecutor.java:81)
    at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteUpdate(GeneralStatement.java:200)
    at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeUpdate(GeneralStatement.java:78)
    at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.update(SqlMapExecutorDelegate.java:505)
    at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.update(SqlMapSessionImpl.java:90)
    at com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.update(SqlMapClientImpl.java:67)

Root Cause:
Oracle archived redo logs used up disk space. Linux 'df' shows the space is 100% used.

Solution:
As the log mentioned, freed space. We did a quick move of archived log files to another disk for the fix. Meanwhile, we are discussing 2 options: either turn on NOARCHIVELOG, or write a script to do automatic gzip and move to other disk.

No comments:

Post a Comment