Fun with applying 19.9 patch set to our test database
Date: Wed, 21 Oct 2020 01:39:40 -0400
Message-ID: <9f5572aa-88c1-3fae-040a-1a4151916534_at_gmail.com>
Like many of my DBA colleagues, I applied the 19.9 release upgrade bundle to our test database. Test database has an active DG standby database, which is mostly used for reporting purposes. I followed the prescribed procedure to the letter, datapatch gave me no trouble, everything seemed fine, but when I checked the standby, there was a nasty surprise:
Recovery interrupted!
stopping change tracking
Errors in file
/opt/oracle/diag/rdbms/stby/ORCLCDB/trace/ORCLCDB_pr00_9069.trc:
ORA-10485: Real-Time Query cannot be enabled while applying migration redo.
2020-10-21 01:07:14.158000 -04:00
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT
NODELAY
Background Media Recovery process shutdown (ORCLCDB)
2020-10-21 01:07:24.944000 -04:00
PL/SQL package SYS.DBMS_BACKUP_RESTORE version is not current
PL/SQL package SYS.DBMS_RCVMAN version 19.08.00.00 is too old
Essentially, standby figured out that DBMS_RCVMAN on the standby side is version behind and decided to stop recovery. However, manual recovery worked as advertised:
ORA-279 signalled during: ALTER DATABASE RECOVER standby database until
cancel ...
2020-10-21 01:10:39.432000 -04:00
rfs (PID:9390): Possible network disconnect with primary database
2020-10-21 01:11:09.807000 -04:00
ALTER DATABASE RECOVER CONTINUE DEFAULT
Media Recovery Log /ora/arch/1_627_1007835883.dbf
Resize operation completed for file# 1, old size 1280000K, new size 1290240K Resize operation completed for file# 1, old size 1290240K, new size 1300480K Resize operation completed for file# 1, old size 1300480K, new size 1310720K Resize operation completed for file# 1, old size 1310720K, new size 1320960K Resize operation completed for file# 1, old size 1320960K, new size 1331200K2020-10-21 01:11:11.043000 -04:00
Resize operation completed for file# 5, old size 440320K, new size 450560K Resize operation completed for file# 5, old size 450560K, new size 460800K Resize operation completed for file# 5, old size 460800K, new size 471040K Resize operation completed for file# 5, old size 471040K, new size 481280K2020-10-21 01:11:12.155000 -04:00
ORA-279 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ... ALTER DATABASE RECOVER CONTINUE DEFAULT Media Recovery Log /ora/arch/1_628_1007835883.dbf 2020-10-21 01:11:14.570000 -04:00
ORA-279 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ... ALTER DATABASE RECOVER CONTINUE DEFAULT Media Recovery Log /ora/arch/1_629_1007835883.dbf 2020-10-21 01:11:16.421000 -04:00
ORA-279 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ... ALTER DATABASE RECOVER CONTINUE DEFAULT Media Recovery Log /ora/arch/1_630_1007835883.dbf 2020-10-21 01:11:20.499000 -04:00
....
ORA-279 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ...
ALTER DATABASE RECOVER CONTINUE DEFAULT
Media Recovery Log /ora/arch/1_634_1007835883.dbf
2020-10-21 01:11:27.594000 -04:00
Resize operation completed for file# 1, old size 1331200K, new size 1341440K
ORA-279 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ...
ALTER DATABASE RECOVER CONTINUE DEFAULT
Media Recovery Log /ora/arch/1_635_1007835883.dbf
Errors with log /ora/arch/1_635_1007835883.dbf
Errors in file
/opt/oracle/diag/rdbms/stby/ORCLCDB/trace/ORCLCDB_pr00_9477.trc:
Linux-x86_64 Error: 2: No such file or directory Additional information: 7
ORA-308 signalled during: ALTER DATABASE RECOVER CONTINUE DEFAULT ... ALTER DATABASE RECOVER CANCEL After that, dgmgrl was able to start recovery and all was well.
DGMGRL for Linux: Release 19.0.0.0.0 - Production on Wed Oct 21 01:25:39
2020
Version 19.9.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.
Welcome to DGMGRL, type "help" for information.
Connected to "ORCLCDB"
Connected as SYSDBA.
DGMGRL> show configuration
Configuration - test
Protection Mode: MaxPerformance Members: orclcdb - Primary database stby - Physical standby database
Fast-Start Failover: Disabled
Configuration Status:
SUCCESS (status updated 55 seconds ago)
DGMGRL> show database stby
Database - stby
Role: PHYSICAL STANDBY Intended State: APPLY-ON Transport Lag: 0 seconds (computed 1 second ago) Apply Lag: 0 seconds (computed 1 second ago) Average Apply Rate: 40.00 KByte/s Real Time Query: OFF Instance(s): ORCLCDB
Database Status:
SUCCESS
So, if you have a standby database, which I know that many of you have,
check the status after the upgrade. If you see that silly message about
DBMS_RCVMAN, do a manual recovery, so that the logs get applied and the
dictionary gets updated. Don't panic. And always have the DBA guide and
a towel close by. You never know when you might run into some Oracle
poetry. BTW, Hitchhiker's Guide Through Galaxy is an excellent book to
re-read during an election season. Of course, I can also recommend some
Marxist literature:
https://www.amazon.com/Groucho-Autobiography-Grouch-Marx-People/dp/B01B7IDL4U#reader_B01B7IDL4U
That should make this depressing time more palatable.
-- Mladen Gogala Database Consultant Tel: (347) 321-1217 -- http://www.freelists.org/webpage/oracle-lReceived on Wed Oct 21 2020 - 07:39:40 CEST