OraFAQ Forum: RAC & Failsafe » crsctl start crs not responding after manually crsctl stop crs

Home » Server Options » RAC & Failsafe » crsctl start crs not responding after manually crsctl stop crs (2 node 11gr2 RAC, OEL 6.4 (3rd Node is configured but downed before ages))

Show: Today's Messages :: Polls :: Message Navigator
E-mail to friend

crsctl start crs not responding after manually crsctl stop crs [message #603400]

Sun, 15 December 2013 23:23

BlackEagle
Messages: 8
Registered: December 2013
Location: India

Junior Member

Hi!

First I have manually stopped CRS:
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'orcl1'
CRS-2673: Attempting to stop 'ora.crsd' on 'orcl1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'orcl1'
CRS-2673: Attempting to stop 'ora.orcl3.vip' on 'orcl1'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'orcl1'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'orcl1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'orcl1'
CRS-2677: Stop of 'ora.orcl3.vip' on 'orcl1' succeeded
CRS-2672: Attempting to start 'ora.orcl3.vip' on 'orcl2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.orcl.db' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'orcl1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'orcl1'
CRS-2676: Start of 'ora.orcl3.vip' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.orcl1.vip' on 'orcl1'
CRS-2677: Stop of 'ora.orcl1.vip' on 'orcl1' succeeded
CRS-2672: Attempting to start 'ora.orcl1.vip' on 'orcl2'
CRS-2676: Start of 'ora.orcl1.vip' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'orcl1'
CRS-2677: Stop of 'ora.asm' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'orcl1'
CRS-2677: Stop of 'ora.ons' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'orcl1'
CRS-2677: Stop of 'ora.net1.network' on 'orcl1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'orcl1' has completed
CRS-2677: Stop of 'ora.crsd' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'orcl1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'orcl1'
CRS-2673: Attempting to stop 'ora.evmd' on 'orcl1'
CRS-2673: Attempting to stop 'ora.asm' on 'orcl1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'orcl1'
CRS-2677: Stop of 'ora.evmd' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.crf' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.asm' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'orcl1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'orcl1'
CRS-2677: Stop of 'ora.cssd' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'orcl1'
CRS-2677: Stop of 'ora.gipcd' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'orcl1'
CRS-2677: Stop of 'ora.gpnpd' on 'orcl1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'orcl1' has completed
CRS-4133: Oracle High Availability Services has been stopped.

At second step I have checked status:
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl check crs
CRS-4639: Could not contact Oracle High Availability Services

At third step I have manually started CRS:
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

At fourth step I have checked CRS status:
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager

At fifth step attempt to shut down once again:
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.

So I thought to do down CRS on all running nodes (Node1 & Node2 are online while Node3 was allready down since ages)
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'orcl1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'orcl1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'orcl1'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'orcl1'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'orcl1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.orcl1.vip' on 'orcl1'
CRS-2677: Stop of 'ora.orcl.db' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'orcl1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'orcl1'
CRS-2677: Stop of 'ora.orcl1.vip' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.crsd' on 'orcl2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'orcl2'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'orcl2'
CRS-2673: Attempting to stop 'ora.orcl.db' on 'orcl2'
CRS-2673: Attempting to stop 'ora.cvu' on 'orcl2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'orcl2'
CRS-2673: Attempting to stop 'ora.orcl2.vip' on 'orcl2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'orcl2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'orcl2'
CRS-2677: Stop of 'ora.orcl2.vip' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'orcl1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'orcl2'
CRS-2677: Stop of 'ora.asm' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'orcl1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.orcl3.vip' on 'orcl2'
CRS-2677: Stop of 'ora.cvu' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.ons' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'orcl1'
CRS-2677: Stop of 'ora.net1.network' on 'orcl1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'orcl1' has completed
CRS-2677: Stop of 'ora.scan1.vip' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.crsd' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'orcl1'
CRS-2673: Attempting to stop 'ora.evmd' on 'orcl1'
CRS-2673: Attempting to stop 'ora.asm' on 'orcl1'
CRS-2677: Stop of 'ora.orcl3.vip' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.orcl.db' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'orcl2'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'orcl2'
CRS-2677: Stop of 'ora.ctssd' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.asm' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'orcl1'
CRS-2677: Stop of 'ora.FRA.dg' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'orcl1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'orcl1'
CRS-2677: Stop of 'ora.cssd' on 'orcl1' succeeded
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'orcl2'
CRS-2677: Stop of 'ora.asm' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'orcl2'
CRS-2677: Stop of 'ora.ons' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'orcl2'
CRS-2677: Stop of 'ora.net1.network' on 'orcl2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'orcl2' has completed
CRS-2677: Stop of 'ora.crsd' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'orcl2'
CRS-2673: Attempting to stop 'ora.evmd' on 'orcl2'
CRS-2673: Attempting to stop 'ora.asm' on 'orcl2'
CRS-2677: Stop of 'ora.evmd' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'orcl2' succeeded
CRS-2677: Stop of 'ora.asm' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'orcl2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'orcl2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'orcl2'
CRS-2677: Stop of 'ora.cssd' on 'orcl2' succeeded

And then started cluster on all nodes from Node1
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl start cluster -all
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'orcl1'
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'orcl2'
CRS-2676: Start of 'ora.cssdmonitor' on 'orcl1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'orcl1'
CRS-2672: Attempting to start 'ora.diskmon' on 'orcl1'
CRS-2676: Start of 'ora.diskmon' on 'orcl1' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'orcl2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'orcl2'
CRS-2672: Attempting to start 'ora.diskmon' on 'orcl2'
CRS-2676: Start of 'ora.diskmon' on 'orcl2' succeeded

And then tried to check CRS status from Node1 and Node2 but still having error:
[root@orcl1 /]# /u01/app/11.2.0/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager

Report message to a moderator

Re: crsctl start crs not responding after manually crsctl stop crs [message #603407 is a reply to message #603400]

Mon, 16 December 2013 00:49

John Watson
Messages: 8962
Registered: January 2010
Location: Global Village

Senior Member

When you copy/paste a terminal session into your post, please enclose it with a [/code] tag at the end and a [code] tag at the beginning.

For how many minutes did you wait between this

/u01/app/11.2.0/grid/bin/crsctl start crs

and

/u01/app/11.2.0/grid/bin/crsctl check crs

Report message to a moderator

Re: crsctl start crs not responding after manually crsctl stop crs [message #603412 is a reply to message #603407]

Mon, 16 December 2013 01:26

BlackEagle
Messages: 8
Registered: December 2013
Location: India

Junior Member

I waited for around a minute and half in between

 /u01/app/11.2.0/grid/bin/crsctl start crs

and

 /u01/app/11.2.0/grid/bin/crsctl check crs

Report message to a moderator

Re: crsctl start crs not responding after manually crsctl stop crs [message #603414 is a reply to message #603412]

Mon, 16 December 2013 01:27

John Watson
Messages: 8962
Registered: January 2010
Location: Global Village

Senior Member

You might want to wait a bit longer.

Report message to a moderator

Re: crsctl start crs not responding after manually crsctl stop crs [message #603415 is a reply to message #603414]

Mon, 16 December 2013 01:29

John Watson
Messages: 8962
Registered: January 2010
Location: Global Village

Senior Member

If you get bored waiting, commands such as

crsctl stat res -init -t

may show you how it is going.

[Updated on: Mon, 16 December 2013 01:29]

Report message to a moderator

Re: crsctl start crs not responding after manually crsctl stop crs [message #603423 is a reply to message #603415]

Mon, 16 December 2013 02:36

BlackEagle
Messages: 8
Registered: December 2013
Location: India

Junior Member

Thanks John!

By waiting for a bit long it is fine.

Many thanks. Smile

Report message to a moderator

Previous Topic:	gsd unknown state
Next Topic:	Difference between activeversion, releaseversion and softwareversion

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Sat Jan 04 16:38:36 CST 2025