Re: RAC cluster down ...

From: laura pena <lizzpenaorclgrp_at_yahoo.com>
Date: Mon, 12 Nov 2007 19:33:04 -0800 (PST)
Message-ID: <733437.38965.qm@web58214.mail.re3.yahoo.com>

Thanks for quick hints.
Here is what I have:
1> I ran crs_stop -all following on the command line prompt:

Attempting to stop `ora.oracle1.LISTENER_INTERGEN_ORACLE1.lsnr` on member `oracle1` Attempting to stop `ora.oracle2.LISTENER_INTERGEN_ORACLE2.lsnr` on member `oracle2` Stop of `ora.oracle1.LISTENER_INTERGEN_ORACLE1.lsnr` on member `oracle1` succeeded. Attempting to stop `ora.oracle1.LISTENER1_ORACLE1.lsnr` on member `oracle1` Stop of `ora.oracle1.LISTENER1_ORACLE1.lsnr` on member `oracle1` succeeded. Attempting to stop `ora.oracle1.vip` on member `oracle1` Stop of `ora.oracle1.vip` on member `oracle1` succeeded. Stop of `ora.oracle2.LISTENER_INTERGEN_ORACLE2.lsnr` on member `oracle2` succeeded. Attempting to stop `ora.oracle2.LISTENER1_ORACLE2.lsnr` on member `oracle2` Stop of `ora.oracle2.LISTENER1_ORACLE2.lsnr` on member `oracle2` succeeded. Attempting to stop `ora.oracle2.vip` on member `oracle2` Stop of `ora.oracle2.vip` on member `oracle2` succeeded.

2> crs_start -all with the following output on command line prompt:

[oracle_at_oracle1 crs]$ crs_start -all
Attempting to start `ora.oracle1.vip` on member `oracle1` Attempting to start `ora.oracle2.vip` on member `oracle2` Start of `ora.oracle2.vip` on member `oracle2` succeeded. Attempting to start `ora.oracle2.LISTENER1_ORACLE2.lsnr` on member `oracle2` Start of `ora.oracle1.vip` on member `oracle1` succeeded. Attempting to start `ora.oracle1.LISTENER1_ORACLE1.lsnr` on member `oracle1` Start of `ora.oracle2.LISTENER1_ORACLE2.lsnr` on member `oracle2` succeeded. Start of `ora.oracle1.LISTENER1_ORACLE1.lsnr` on member `oracle1` succeeded. Attempting to start `ora.oracle2.LISTENER_INTERGEN_ORACLE2.lsnr` on member `oracle2` Attempting to start `ora.oracle1.LISTENER_INTERGEN_ORACLE1.lsnr` on member `oracle1` Start of `ora.oracle2.LISTENER_INTERGEN_ORACLE2.lsnr` on member `oracle2` succeeded. Start of `ora.oracle1.LISTENER_INTERGEN_ORACLE1.lsnr` on member `oracle1` succeeded.

3> starting instance powersv2 (I have another instance as well to start intergen, is there a way to start both?)

[oracle_at_oracle1 crs]$ srvctl start instance -d powersv2 PRKO-2001 : Invalid command line syntax
[oracle_at_oracle1 crs]$ srvctl start database -d powersv2 PRKP-1001 : Error starting instance powersv21 on node oracle1 CRS-1028: Dependency analysis failed because of: CRS-0223: Resource 'ora.powersv2.powersv21.inst' has placement error. PRKP-1001 : Error starting instance powersv22 on node oracle2 CRS-1028: Dependency analysis failed because of: CRS-0223: Resource 'ora.powersv2.powersv22.inst' has placement error.

crsd.log file shows:
2007-11-12 20:16:10.713: [ CRSRES][1577253184]0CRS-1028: Dependency analysis failed because of: 'Resource in UNKNOWN state: ora.powersv2.powersv21.inst'

2007-11-12 20:16:10.717: [ CRSRES][1545783616]0CRS-1028: Dependency analysis failed because of: 'Resource in UNKNOWN state: ora.powersv2.powersv22.inst'

4> the only log file I see for powersv21.log hasn't been touched since sept 26 [oracle_at_oracle1 10gR2]$ pwd
/home/oracle/product/10gR2
[oracle_at_oracle1 10gR2]$ tail ./log/oracle1/racg/ora.powersv2.powersv21.inst.log Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved. 2007-09-26 20:44:13.784: [ RACG][2899849808] [2355][2899849808][ora.default]: Cannot read string from /home/oracle/product/10gR2/racg/init/oracle1/powersv2/.connect

If I try to bring up manually using sqlplus /nolog : Is works... but I only brought up where ORACLE_SID = powersv21, so really I don't have it up right (sorry I'm just a novice, I obviously do not understand some basics here)?

[oracle_at_oracle1 10gR2]$ export ORACLE_SID=powersv21 [oracle_at_oracle1 10gR2]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.3.0 - Production on Mon Nov 12 20:30:37 2007

SQL> connect sys as sysdba
Enter password:
Connected to an idle instance.
SQL> startup
ORACLE instance started.

Total System Global Area 1610612736 bytes

Fixed Size                  2073088 bytes
Variable Size             553651712 bytes
Database Buffers         1040187392 bytes
Redo Buffers               14700544 bytes

Database mounted.
Database opened.

Again any help would be greatly appreciated.

Many Thanks.
Alex Gorbachev <ag_at_oracloid.com> wrote: Could you provide the output of $ORA_CRS_HOME/bin/crs_stat ? You said you use OCFS2 so no ASM. Right?

What's in the logs $ORA_CRS_HOME/log//crsd/crsd.log ( is the node name - check both nodes).

What's in the logs
$ORACLE_HOME/log//racg/ora.powersv2.powersv21.inst.log (and ora.powersv2.powersv22.inst.log).

Without srvctl - try to set your environments (ORACLE_HOME, ORACLE_SID and etc) and startup using SQL*Plus.

On Nov 12, 2007 6:27 PM, laura pena
wrote:
> So we rebooted the nodes and now are unable to bring up RAC cluster.
> Prior the cluster was up for 3 weeks.
>
> OS: Enterprise Linux 2.6.18-8.1.8.el5 x86_64 (64bit)
> Oracle: using ocfs2 (version2.0) on 10.2.0.3
>
> Getting the following error trying to start....
>
> srvctl start database -d powersv2
>
> PRKP-1001: Error starting instance powersv21 on node oracle1
> CRS-1028: Dependency analysis failed because of:
> CRS-0223: Resource: 'ora.powersv2.powersv21.inst' has placement error.
> PRKP-1001: Error starting instance powersv22 on node oracle2
> CRS-1028: Dependency analysis failed because of:
> CRS--0223: Resource 'ora.powersv2.powersv22.inst' has placement error.
>
>
> Any ideas would be greatly appreciated while I wait for Oracle to respond.
>
> Thanks.
>
>
>
>
>
>
> ________________________________
> Be a better pen pal. Text or chat with friends inside Yahoo! Mail. See how.

-- 
Alex Gorbachev, Oracle DBA Brewer, The Pythian Group
http://www.pythian.com/blogs/author/alex http://www.oracloid.com
BAAG party - www.BattleAgainstAnyGuess.com


       
---------------------------------
Never miss a thing.   Make Yahoo your homepage.
--
http://www.freelists.org/webpage/oracle-l

Received on Mon Nov 12 2007 - 21:33:04 CST