ORA-15003 on one node of RAC with ASM

From: saurabh manroy <saurabhmanroy_at_gmail.com>
Date: Wed, 23 Oct 2013 17:54:39 +0200
Message-ID: <CAELkjfU0p4aZtznuYn-zxKFzFLAvvYfSuDFCOC2VwifgDjQRVA_at_mail.gmail.com>

Listers,
Here is the problem that we have in one of our production DB and an Oracle SR is also open for this:
DB/CRS/ASM : 11.1.0.7
Early morning today, RDBMS and ASM instances on one node of a two node cluster went down. Later, while trying to restart ASM instance on another node, it reported following errors while trying to mount diskgroups on failed node in the alert log.

ERROR: diskgroup xxxx was not mounted
ERROR: PST found another heartbeat (grp 2)

On the console
ORA-15032: not all alterations performed ORA-15003: diskgroup "xxxx" already mounted in another lock name space

Looking at crs_stat -t output shows that each node can't see resources from another node. For example:

On Node1:
ora.node1.vip application ONLINE ONLINE node1 ora....SM2.asm application ONLINE OFFLINE

Node2:
ora.node1.vip application ONLINE OFFLINE ora....SM2.asm application ONLINE ONLINE node2

Oracle support has suggested to perform crs restart on node2 and if it doesn't help a reboot of complete cluster which we'll perform tonight. Has anybody seen this before ? What might be the root cause of ORA-15003 error on RAC instance?

Regards,
Saurabh Manroy

--
http://www.freelists.org/webpage/oracle-l

Received on Wed Oct 23 2013 - 17:54:39 CEST