RAConfused!!!

Hello RAC people,  

RAC 10.1.0.3 Tru64 5.1

TruCluster and CRS  

We have a 4 node 10.1.03 RAC cluster on Tru64 5.1b with CRS and TruCluster. During the last (and the only one after the CRS installation) CRS shutdown (not reboot), listener on 2 nodes came back up automatically and not on the other 2. The order of shutting down CRS was:

NODE NAME:   Time                 Order
Node1               3:48pm             1
Node2               3:54pm             2
Node3               4:00:01pm         3
Node4               4:00:17pm         4

Listener on node3 and node4 came back up at 4:00:36pm and 4:00:41pm respectively.
I always thought that the expected behavior is when you shutdown CRS, it will bring down all the services/instances/listeners registered and will not restart automatically "before you start CRS". If I am correct, why the listeners on node3 and node4 did come back up and if I am wrong, why the listeners on node1 and node2 didn't come up fine.  

I looked at the following files, but couldn't find anything too relevant:

CRS_HOME/crs/log/*
CRS_HOME/crs/crs/log/*
CRS_HOME/crs/crs/init/*
CRS_HOME/crs/crs/public/*
CRS_HOME/crs/css/log/*
CRS_HOME/crs/css/init/*
CRS_HOME/crs/evm/log/*
CRS_HOME/crs/evm/init/*
CRS_HOME/crs/srvm/log/*
CRS_HOME/crs/srvm/init/*
CRS_HOME/crs/racg/dump/*

The only thing that I found unusual in crs logs was:  

Node1.log (similar in Node2.log):  

`ora.node1.ons` ran out of restarts on `node1`

`ora.node1.ons` failed on `node1`, relocating.

Cannot relocate `ora.node1.ons`. Stopping dependents  

Node3.log (similar in Node4.log): No entry like above!!  

Any ideas? Is "ONS" (and running ONSCA) "mandatory" and not "optional"? BTW, ONSCA did finish successfully at the time of installation.  

TIA, -Amit.  

--
http://www.freelists.org/webpage/oracle-l