Oracle 11.0.2.4 EE failed to start, possible crsd problem

From: Chen Zhou <oracle.unknowns_at_gmail.com>
Date: Mon, 13 Jun 2016 10:45:35 -0700
Message-ID: <CAJUY3dTO7gDQB8On+NGD8GFk7XZ61zj+eEoNH4xWK46Rqhy_zQ_at_mail.gmail.com>



Hi, Everyone,

This is a stand-alone 11.2.0.4 EE database on GI/ASM. It has been applied with PSU patches regularly with no issue. At the time it had Jan 2016 PSU on both GI and OHOME.
When I tried to apply the Apr 2016 PSU with the same "opatch auto" command, it failed with this message right away before it started patching:

Using configuration parameter file: $GI_HOME/crs/install/crsconfig_params Not able to retreive database home information

(Yes, "retreive" was spelled wrong.)

So I tried to stop the database with "srvctl stop database -d", it failed too:

PRCD-1027 : Failed to retrieve database XYZ PRCR-1115 : Failed to find entities of type resource that match filters ((NAME == ora.xyz.db) && (TYPE == ora.database.type)) and contain attributes VERSION,ORACLE_HOME,DATABASE_TYPE Cannot communicate with crsd

Then I tried sqlplus> shutdown immediate. It hanged. Tried shutdown abort. Then just wanted to startup to see if it could. It couldn't. There were error messages in ASM:

ORA-29786: SIHA attribute GET failed with error [Attribute 'ASM_DISKSTRING' sts[184] lsts[0]]

But there had not been any changes to the database or ASM since last PSU patch. The diskstring has been the same and all devices are there with the correct ownership/permission.

So SA had to power off the server and power it on. After that, ASM started back fine and the database has been running normal so far.

I opened a Oracle Sev 1 ticket right when the database got stuck starting up, that was Saturday morning. I even called to escalate the issue and talked with a support manager after wasting 50 minutes waiting. 3 support engineers took brief turns asking me to provide logs/info and went off their shifts. So far I still have no feedback on what might have gone wrong and next time we attempt patching it again if we will have the same issue.

Do you have any idea what have gone wrong? Any suggestion on checking crsd?

Thanks a bundle,

Chen

--
http://www.freelists.org/webpage/oracle-l
Received on Mon Jun 13 2016 - 19:45:35 CEST

Original text of this message