Oracle-L: RE: Raid Arrays and Power Loss

From: Igor Neyman <ineyman_at_perceptron.com>
Date: Tue, 16 Sep 2003 05:49:24 -0800
Message-ID: <F001.005D00AD.20030916054924@fatcity.com>

Ian,

Thanks for sharing (seriously).

Igor Neyman, OCP DBA
ineyman_at_perceptron.com

-----Original Message-----
MacGregor, Ian A.
Sent: Monday, September 15, 2003 11:34 PM To: Multiple recipients of list ORACLE-L

Last Friday was hot here, and rumor has it our 230 KV power line sagged and touched some tree branches. The local power company shut it off. Leaving our systems to depend on UPS. About 30 minutes afterwards one system produced these errors. This was jus before the system went dead

Fri Sep 12 12:58:40 2003
Errors in file /opt/oracle/admin/BBRO/bdump/bbro_ckpt_1420.trc:

ORA-00206: error in writing (block 3, # blocks 1) of controlfile
ORA-00202: controlfile: '/u1/oradata/BBRO/BBROcntrl01.ctl'
ORA-27063: skgfospo: number of bytes read/written is incorrect

SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 8192
Fri Sep 12 12:58:42 2003
Errors in file /opt/oracle/admin/BBRO/bdump/bbro_ckpt_1420.trc:

ORA-00221: error on write to controlfile
ORA-00206: error in writing (block 3, # blocks 1) of controlfile
ORA-00202: controlfile: '/u1/oradata/BBRO/BBROcntrl01.ctl'
ORA-27063: skgfospo: number of bytes read/written is incorrect

SVR4 Error: 5: I/O error
Additional information: -1
Additional information: 8192
Fri Sep 12 12:58:42 2003
CKPT: terminating instance due to error 221 Instance terminated by CKPT, pid = 1420

Things look pretty shaky here. When things were restarted the following error was produced.
Fri Sep 12 13:32:01 2003

ORA-00204: error in reading (block 1, # blocks 1) of controlfile
ORA-00202: controlfile: '/u1/oradata/BBRO/BBROcntrl01.ctl'
ORA-27091: skgfqio: unable to queue I/O

SVR4 Error: 6: No such device or address Additional information: 1

The raid array had not been powered on

However
Fri Sep 12 15:33:08 2003
ORA-00202: controlfile: '/u1/oradata/BBRO/BBROcntrl01.ctl' ORA-27037: unable to obtain file status
SVR4 Error: 2: No such file or directory Additional information: 3
Fri Sep 12 15:33:11 2003
ORA-205 signalled during: alter database mount...

Now the file system is available, but the file itself has disappeared. It was not corrupted, just disappeared. We duplex a copy to an internal disk. So recovery was easy.

However once this was fixed

Fri Sep 12 16:18:58 2003
Thread recovery: start rolling forward thread 1 Fri Sep 12 16:18:58 2003
Errors in file /opt/oracle/admin/BBRO/udump/bbro_ora_1804.trc:

ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/u2/oradata/BBRO/redo0301.log'
ORA-27037: unable to obtain file status

SVR4 Error: 2: No such file or directory Additional information: 3

ORA-313 signalled during: ALTER DATABASE OPEN...
------------------------------------------------------------------------
-------------------------------------

These files are on a RAID 1 LUN. Both copies of the file are gone. Again not corrupted but gone. I don't know if using duplexing rather than RAID 1 would have mattered here, but I am changing things so that one group of redo logs is on internal disk and written via the duplexing method.

Ian MacGregor
Stanford linear Accelerator Center
ian_at_SLAC.Stanford.edu

--

Please see the official ORACLE-L FAQ: http://www.orafaq.net
--

Author: MacGregor, Ian A.
INET: ian_at_SLAC.Stanford.EDU

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services
---------------------------------------------------------------------

To REMOVE yourself from this mailing list, send an E-Mail message to: ListGuru_at_fatcity.com (note EXACT spelling of 'ListGuru') and in the message BODY, include a line containing: UNSUB ORACLE-L (or the name of mailing list you want to be removed from). You may also send the HELP command for other information (like subscribing).

--

Please see the official ORACLE-L FAQ: http://www.orafaq.net
--

Author: Igor Neyman
INET: ineyman_at_perceptron.com

Fat City Network Services    -- 858-538-5051 http://www.fatcity.com
San Diego, California        -- Mailing list and web hosting services
---------------------------------------------------------------------