| Inconsistent Dictionary on Standby after Reinstatement [message #531813] |
Thu, 17 November 2011 15:19  |
 |
tolq
Messages: 1
Registered: November 2011
Location: Calgary, Canada
|
Junior Member |
|
|
Hi,
My problems (as I understand it) is that after successful failover and successful reinstatement of new standby
database, redo logs do not applied to standby database because. Value on RESETLOGS_CHANGE#
is inconsistent on Standby database.
There are details:
1. Configuration
Server smphoracledb01 with Primary database DB_UNIQUE_NAME = NVCSOM1A
Server smphoracledb02 with Standby database DB_UNIQUE_NAME = NVCSOM1B
Server smphoracledb03 with Data Guard Observer running.
All servers run Linux with Oracle 11.2.0.1.0.
2. Server with Primary database failed and database failed over to Standby server.
No problem here. Primary database NVCSOM1B is up and running on server smphoracledb02.
3. After server smphoracledb01 was fixed, I started database NVCSOM1A 'nomount'. I expected
that it will be reinstate automatically by Observer, but it did not happen. I checked that
Observer can connect to both servers (Primary and Standby).
4. I've started DGMGRL on current Primary server smphoracledb02 and check configuration.
$> dgmgrl sys/xxxx
DGMGRL for Linux: Version 11.2.0.1.0 - 64bit Production
Copyright (c) 2000, 2009, Oracle. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Connected.
DGMGRL> show configuration
Configuration - nvcsom1
Protection Mode: MaxPerformance
Databases:
nvcsom1b - Primary database
Warning: ORA-16829: fast-start failover configuration is lagging
nvcsom1a - (*) Physical standby database (disabled)
ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: ENABLED
Configuration Status:
WARNING
5. I started manual reinstatement and it succeeded. But configuration is still broken.
DGMGRL> reinstate database nvcsom1a;
Reinstating database "nvcsom1a", please wait...
Operation requires shutdown of instance "nvcsom1" on database "nvcsom1a"
Shutting down instance "nvcsom1"...
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "nvcsom1" on database "nvcsom1a"
Starting instance "nvcsom1"...
ORACLE instance started.
Database mounted.
Continuing to reinstate database "nvcsom1a" ...
Reinstatement of database "nvcsom1a" succeeded
DGMGRL> show configuration
Configuration - nvcsom1
Protection Mode: MaxPerformance
Databases:
nvcsom1b - Primary database
Error: ORA-16810: multiple errors or warnings detected for the database
nvcsom1a - (*) Physical standby database
Warning: ORA-16829: fast-start failover configuration is lagging
Fast-Start Failover: ENABLED
Configuration Status:
ERROR
6. Data Guard log file shows only these errors:
2011-11-14 08:55:10.944 RSM0: HEALTH CHECK WARNING: ORA-16829: fast-start failover configuration is lagging
2011-11-14 08:55:10.944 Operation HEALTH_CHECK canceled during phase 2, error = ORA-16810
2011-11-14 08:56:10.809 RSM0: HEALTH CHECK ERROR: ORA-16783: cannot resolve gap for database nvcsom1a
2011-11-14 08:56:10.809 Found unresolvable gap to database nvcsom1a.
7. From SQL*Plus on SMPHORACLEDB02 database NVCSOM1B (current Primary) I found
the problem. Archive redo logs were shipped to Standby but not applied:
SYS@nvcsom1> select db_unique_name, database_role, open_mode from v$database;
DB_UNIQUE_NAME DATABASE_ROLE OPEN_MODE
------------------------------ ---------------- --------------------
nvcsom1b PRIMARY READ WRITE
SYS@nvcsom1> select max(resetlogs_change#) from v$archived_log;
MAX(RESETLOGS_CHANGE#)
----------------------
9999251
SYS@nvcsom1> alter system switch logfile;
System altered.
SYS@nvcsom1> select sequence#, archived, applied, deleted, standby_dest
from v$archived_log
where resetlogs_change# = 9999251
order by 1,5;
SEQUENCE# ARC APPLIED DEL STA
---------- --- --------- --- ---
1 YES NO YES NO
2 YES NO YES NO
3 YES NO YES NO
3 YES NO NO YES
4 YES NO YES NO
4 YES NO NO YES
5 YES NO YES NO
5 YES NO NO YES
6 YES NO YES NO
6 YES NO NO YES
7 YES NO YES NO
7 YES NO NO YES
8 YES NO YES NO
8 YES NO NO YES
8. Investigation shows that probable cause of the problems is that after reinstatement new
Standby database has inconsistent information about resetlogs change number.
These are results from Standby:
SYS@nvcsom1> select resetlogs_change# from v$database;
RESETLOGS_CHANGE#
-----------------
773741
SYS@nvcsom1> select max(resetlogs_change#) from v$archived_log;
MAX(RESETLOGS_CHANGE#)
----------------------
9999251
SYS@nvcsom1> select RESETLOGS_CHANGE#, RESETLOGS_TIME, PRIOR_RESETLOGS_CHANGE#, PRIOR_RESETLOGS_TIME from v$database;
RESETLOGS_CHANGE# RESETLOGS PRIOR_RESETLOGS_CHANGE# PRIOR_RES
----------------- --------- ----------------------- ---------
773741 30-AUG-11 773166 30-AUG-11
SYS@nvcsom1> SELECT RESETLOGS_CHANGE# FROM V$DATABASE_INCARNATION WHERE STATUS = 'CURRENT';
RESETLOGS_CHANGE#
-----------------
9999251
9. The same queries on Primary database return these:
SYS@nvcsom1> select resetlogs_change# from v$database;
RESETLOGS_CHANGE#
-----------------
9999251
SYS@nvcsom1> select max(resetlogs_change#) from v$archived_log;
MAX(RESETLOGS_CHANGE#)
----------------------
9999251
SYS@nvcsom1> select RESETLOGS_CHANGE#, RESETLOGS_TIME, PRIOR_RESETLOGS_CHANGE#, PRIOR_RESETLOGS_TIME from v$database;
RESETLOGS_CHANGE# RESETLOGS PRIOR_RESETLOGS_CHANGE# PRIOR_RES
----------------- --------- ----------------------- ---------
9999251 13-NOV-11 773741 30-AUG-11
SYS@nvcsom1> SELECT RESETLOGS_CHANGE# FROM V$DATABASE_INCARNATION WHERE STATUS = 'CURRENT';
RESETLOGS_CHANGE#
-----------------
9999251
=========================================
To fix the problem I have recreated standby database.
Now I have two questions:
1. Why did it happen?
2. Is there the better way to fix the standby database?
Thank you,
Toli.
|
|
|
|
|
|
] 
Blog
Search
Help
Members
Register
Login
Home
