July22 Details:
===============


Tue Jul 22 14:29:28 2008
Thread 2 advanced to log sequence 541
  Current log# 3 seq# 541 mem# 0: +ASMPROD/abcd/onlinelog/group_3.274.603069543
  Current log# 3 seq# 541 mem# 1: +ASMFLASH/abcd/onlinelog/group_3.262.603069563
Tue Jul 22 23:34:33 2008
ALTER SYSTEM SET service_names='abcd' SCOPE=MEMORY SID='abcd2';
Tue Jul 22 23:34:34 2008
ALTER SYSTEM SET service_names='abcd','abcd_CON' SCOPE=MEMORY SID='abcd2';

=======================================
Tue Jul 22 23:34:35 2008
Shutting down instance (abort)
License high water mark = 343
Instance terminated by USER, pid = 27875
========================================

Tue Jul 22 23:34:51 2008
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 lan1 10.10.0.0 configured from OCR for use as a cluster interco
nnect
Interface type 1 lan3 10.1.0.0 configured from OCR for use as  a public interfac
e
Picked latch-free SCN scheme 1
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
  processes                = 500
  sessions                 = 555
  resource_limit           = TRUE
  sga_max_size             = 3221225472
  __shared_pool_size       = 402653184
  __large_pool_size        = 100663296
  __java_pool_size         = 16777216
  __streams_pool_size      = 0
  spfile                   = +ASMPROD/abcd/spfileabcd.ora
  sga_target               = 3221225472
  control_files            = +ASMPROD/abcd/controlfile/current.265.603067087, +A
SMFLASH/abcd/controlfile/current.259.603067091
  db_block_size            = 8192
  __db_cache_size          = 2684354560
  db_writer_processes      = 4
  compatible               = 10.2.0.1.0
  log_archive_dest_1       = LOCATION=+ASMFLASH/abcd/
  log_archive_format       = %t_%s_%r.dbf
  db_file_multiblock_read_count= 16
  cluster_database         = TRUE
  cluster_database_instances= 2
  db_create_file_dest      = +ASMPROD
  db_recovery_file_dest    = +ASMFLASH
  db_recovery_file_dest_size= 64424509440
  thread                   = 2
  instance_number          = 2
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS2
  remote_login_passwordfile= EXCLUSIVE
  db_domain                =
  dispatchers              = (protocol=TCP)(listener=listeners_abcd)
  shared_servers           = 2
  remote_listener          = LISTENERS_abcd
  job_queue_processes      = 10
  background_dump_dest     = /u01/app/oracle/admin/abcd/bdump
  user_dump_dest           = /u01/app/oracle/admin/abcd/udump
  core_dump_dest           = /u01/app/oracle/admin/abcd/cdump
  audit_file_dest          = /u01/app/oracle/admin/abcd/adump
  db_name                  = abcd
  open_cursors             = 3000
  optimizer_mode           = CHOOSE
  pga_aggregate_target     = 847249408
Cluster communication is configured to use the following interface(s) for this i
nstance
  10.10.10.2
Tue Jul 22 23:34:59 2008
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=28463
DIAG started with pid=3, OS id=28465
PSP0 started with pid=4, OS id=28489
LMON started with pid=5, OS id=28496
LMD0 started with pid=6, OS id=28498
LMS0 started with pid=7, OS id=28500
LMS1 started with pid=8, OS id=28502
MMAN started with pid=9, OS id=28504
DBW0 started with pid=10, OS id=28506
DBW1 started with pid=11, OS id=28508
DBW2 started with pid=12, OS id=28520
DBW3 started with pid=13, OS id=28535
LGWR started with pid=14, OS id=28541
CKPT started with pid=15, OS id=28543
SMON started with pid=16, OS id=28545
RECO started with pid=17, OS id=28547
CJQ0 started with pid=18, OS id=28549
MMON started with pid=19, OS id=28551
Tue Jul 22 23:35:02 2008
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL
=TCP))'...
MMNL started with pid=20, OS id=28553
Tue Jul 22 23:35:02 2008
starting up 2 shared server(s) ...
Tue Jul 22 23:35:05 2008
lmon registered with NM - instance id 2 (internal mem no 1)
Tue Jul 22 23:35:06 2008
Reconfiguration started (old inc 0, new inc 8)
pseudo shared rm latch used
List of nodes:
 0 1
 Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
 Communication channels reestablished
 * domain 0 valid = 1 according to instance 0
Tue Jul 22 23:35:07 2008
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Tue Jul 22 23:35:07 2008
 LMS 0: 0 GCS shadows cancelled, 0 closed
Tue Jul 22 23:35:07 2008
 LMS 1: 0 GCS shadows cancelled, 0 closed
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
Tue Jul 22 23:35:08 2008
 LMS 0: 0 GCS shadows traversed, 0 replayed
Tue Jul 22 23:35:08 2008
 LMS 1: 0 GCS shadows traversed, 0 replayed
Tue Jul 22 23:35:08 2008
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=24, OS id=28850
Tue Jul 22 23:35:15 2008
  db_name                  = abcd
  open_cursors             = 3000
  optimizer_mode           = CHOOSE
  pga_aggregate_target     = 847249408
Cluster communication is configured to use the following interface(s) for this i
nstance
  10.10.10.2
Tue Jul 22 23:34:59 2008
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=28463
DIAG started with pid=3, OS id=28465
PSP0 started with pid=4, OS id=28489
LMON started with pid=5, OS id=28496
LMD0 started with pid=6, OS id=28498
LMS0 started with pid=7, OS id=28500
LMS1 started with pid=8, OS id=28502
MMAN started with pid=9, OS id=28504
DBW0 started with pid=10, OS id=28506
DBW1 started with pid=11, OS id=28508
DBW2 started with pid=12, OS id=28520
DBW3 started with pid=13, OS id=28535
LGWR started with pid=14, OS id=28541
CKPT started with pid=15, OS id=28543
SMON started with pid=16, OS id=28545
RECO started with pid=17, OS id=28547
CJQ0 started with pid=18, OS id=28549
MMON started with pid=19, OS id=28551
Tue Jul 22 23:35:02 2008
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL
=TCP))'...
MMNL started with pid=20, OS id=28553
Tue Jul 22 23:35:02 2008
starting up 2 shared server(s) ...
Tue Jul 22 23:35:05 2008
lmon registered with NM - instance id 2 (internal mem no 1)
Tue Jul 22 23:35:06 2008
Reconfiguration started (old inc 0, new inc 8)
pseudo shared rm latch used
List of nodes:
 0 1
 Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
 Communication channels reestablished
 * domain 0 valid = 1 according to instance 0
Tue Jul 22 23:35:07 2008
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Tue Jul 22 23:35:07 2008
 LMS 0: 0 GCS shadows cancelled, 0 closed
Tue Jul 22 23:35:07 2008
 LMS 1: 0 GCS shadows cancelled, 0 closed
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
Tue Jul 22 23:35:08 2008
 LMS 0: 0 GCS shadows traversed, 0 replayed
Tue Jul 22 23:35:08 2008
 LMS 1: 0 GCS shadows traversed, 0 replayed
Tue Jul 22 23:35:08 2008
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=24, OS id=28850
Tue Jul 22 23:35:15 2008
UCCESS: diskgroup ASMFLASH was mounted
ue Jul 22 23:35:28 2008
etting recovery target incarnation to 1
ue Jul 22 23:35:28 2008
uccessful mount of redo thread 2, with mount id 1821067761
ue Jul 22 23:35:28 2008
atabase mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
ompleted: ALTER DATABASE   MOUNT
ue Jul 22 23:35:30 2008
LTER DATABASE OPEN
icked broadcast on commit scheme to generate SCNs
ue Jul 22 23:35:33 2008
GWR: STARTING ARCH PROCESSES
RC0 started with pid=31, OS id=29318
ue Jul 22 23:35:33 2008
RC0: Archival started
RC1 started with pid=32, OS id=29320
ue Jul 22 23:35:33 2008
RC1: Archival started
GWR: STARTING ARCH PROCESSES COMPLETE
hread 2 opened at log sequence 542
 Current log# 4 seq# 542 mem# 0: +ASMPROD/abcd/onlinelog/group_4.275.603069581
 Current log# 4 seq# 542 mem# 1: +ASMFLASH/abcd/onlinelog/group_4.263.603069603
uccessful open of redo thread 2
ue Jul 22 23:35:34 2008
TTR advisory is disabled because FAST_START_MTTR_TARGET is not set
ue Jul 22 23:35:34 2008
RC0: STARTING ARCH PROCESSES
ue Jul 22 23:35:34 2008
RC1: Becoming the 'no FAL' ARCH
RC1: Becoming the 'no SRL' ARCH
ue Jul 22 23:35:34 2008
MON: enabling cache recovery
ue Jul 22 23:35:34 2008
RC2: Archival started
RC0: STARTING ARCH PROCESSES COMPLETE
RC0: Becoming the heartbeat ARCH
RC2 started with pid=33, OS id=29322
ue Jul 22 23:35:39 2008
uccessfully onlined Undo Tablespace 4.
ue Jul 22 23:35:39 2008
MON: enabling tx recovery
ue Jul 22 23:35:39 2008
atabase Characterset is AR8ISO8859P6
eplication_dependency_tracking turned off (no async multimaster replication fou
d)
tarting background process QMNC
MNC started with pid=36, OS id=29584
ue Jul 22 23:35:48 2008
ompleted: ALTER DATABASE OPEN
ue Jul 22 23:41:33 2008
hutting down archive processes
ue Jul 22 23:41:38 2008
RCH shutting down
RC2: Archival stopped
ed 

Jul 23 Details:
================
Jul 23 08:06:25 2008
hread 2 advanced to log sequence 543
 Current log# 3 seq# 543 mem# 0: +ASMPROD/abcd/onlinelog/group_3.274.603069543
 Current log# 3 seq# 543 mem# 1: +ASMFLASH/abcd/onlinelog/group_3.262.603069563

ed Jul 23 09:52:07 2008
emory Notification: Library Cache Object loaded into SGA
eap size 2073K exceeds notification threshold (2048K)
etails in trace file /u01/app/oracle/admin/abcd/bdump/nprd2_ora_28584.trc
GL object name :    SELECT count(*) INTO :num_valid_specs    FROM  all_objects
  WHERE object_name = 'DBMS_DEBUG'    AND   owner = 'SYS'    AND   object_type
 'PACKAGE'    AND   status = 'VALID'