Friday, September 17, 2010

Tomcat is not responding due to ORA-00257

Web team reports that request to tomcat is not responding all of a sudden. The environment is an isolated one with access control, which means there is no system or application change recently.

Application Log:
Caused by: com.inet.ora.Ora4SQLException: [Oracle] #1 ORA-00257: archiver error. Connect internal only, until freed.
[Oracle] #1
    at com.inet.ora.i.a(Unknown Source)
    at com.inet.ora.OraConnection.a(Unknown Source)
    at com.inet.ora.OraConnection.q(Unknown Source)
    at com.inet.ora.OraConnection.<init>(Unknown Source)
    at com.inet.ora.l.<init>(Unknown Source)
    at com.inet.ora.i.a(Unknown Source)
    at com.inet.ora.OraDriver.connect(Unknown Source)
    at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(
    at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(
    at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(
    at org.apache.commons.dbcp.BasicDataSource.createDataSource(
    ... 46 more

Thread Dump:
"http-8080-12" daemon prio=10 tid=0x0000000046b20c00 nid=0x18e0 runnable [0x000000004954e000..0x000000004954fb90]
   java.lang.Thread.State: RUNNABLE
    at Method)
    at com.inet.ora.OraConnection.i(Unknown Source)
    at com.inet.ora.OraConnection.a(Unknown Source)
    - locked <0x00002aab89aad9d0> (a com.inet.ora.l)
    at Source)
    at Source)
    at Source)
    at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(
    at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeUpdate(
    at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteUpdate(
    at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeUpdate(
    at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.update(
    at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.update(
    at com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.update(

Root Cause:
Oracle archived redo logs used up disk space. Linux 'df' shows the space is 100% used.

As the log mentioned, freed space. We did a quick move of archived log files to another disk for the fix. Meanwhile, we are discussing 2 options: either turn on NOARCHIVELOG, or write a script to do automatic gzip and move to other disk.

No comments:

Post a Comment