Skip navigation.

Syed Jaffar

Syndicate content
Whatever topic has been discussed on this blog is my own finding and views, not necessary match with others. I strongly recommend you to do a test before you implement the piece of advice given at my blog.The Human Flyhttp://www.blogger.com/profile/03489518270084955004noreply@blogger.comBlogger218125
Updated: 7 hours 32 min ago

Oracle 11204 Clusterware upgrade - ASM glitch

Tue, 2014-08-19 03:01
Yet another tough challenge thrown at my team right after the disaster recovery (DR) simulation drill which performed barely couple of weeks ago. The new task (challenge) in hands is to upgrade the existing four cluster environments from 11.2.0.2 to 11.2.0.4 as Oracle already stopped supporting v11.2.0.2.

Although last week we had a 3 node successfully upgrade track record, we encountered ASM upgrade troubles whilst running rootupgrade.sh in a new cluster environment (7 nodes). The following error was reported during the course of rootupgrade.sh script execution:

CRS-2672: Attempting to start 'ora.asm' on 'node01' 
CRS-5017: The resource action "ora.asm start" encountered the following error: 
ORA-48108: invalid value given for the diagnostic_dest init.ora parameter 
ORA-48140: the specified ADR Base directory does not exist [/u00/app/11.2.0/grid/dbs/{ORACLE_BASE}] 
ORA-48187: specified directory does not exist 
HPUX-ia64 Error: 2: No such file or directory 
Additional information: 1CRS-2674: Start of 'ora.asm' on 'node01' failed 
CRS-2679: Attempting to clean 'ora.asm' on 'node01' 
CRS-2681: Clean of 'ora.asm' on 'node01' succeeded 
CRS-4000: Command Start failed, or completed with errors. 

When tried to start-up the ASM instance manually through sqlplus prompt, the following error was thrown:

SQL> 
ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance 
ORA-48108: invalid value given for the diagnostic_dest init.ora parameter 
ORA-48140: the specified ADR Base directory does not exist [/u00/app/11.2.0/grid/dbs/{ORACLE_BASE}] 
ORA-48187: specified directory does not exist 
HPUX-ia64 Error: 2: No such file or directory

Sadly, there wasn't much info available about the nature of this problem. As usual, after giving it 1 hr try with different options, we opened a SR with Oracle support and agreed to rollback the upgrade from the node where the rootupgrade script failed. Luckily, this was the first node we tried and other 6 nodes were just running fine. After rolling back to the previous cluster version, ASM instance error was still persist.

To resolve the ASM instance startup issues, the following action was taken:

  • export diagnostic_dest=/u00/app/oracle
  • From active ASM instance on another node, executed the following statement:
    • SQL> ALTER SYSTEM STOP ROLLING MIGRATION;

Cause:The problem caused an ASM instance startup issue was reported/logged as a known bug (17449823).
Workaround:According to the MOS Doc ID (1598959.1), the bug is still being worked by the development team, they suggest the following work around on each node just before running the rootupgrade.sh script:
  • mkdir <New-GI-HOME>/dbs/{ORACLE_BASE} 
Third successful attemptThe upgrade failed in first 2 attempts, and the 3 attempt was successful and we managed to upgrade all 7 nodes from 11.2.0.2 to 11.2.0.4. It was also learnt that CRS_HOME, ORACLE_HOME, ORACLE_BASE was not unset before the runinstaller was initiated. In 3rd attempt with unsetting those parameters, upgrade went successfully.
Addendum (24-Aug-2014)
Couple of new challenges encountered in the last  upgrade task on 10 nodes.

  1. OUI window from which runInstaller was initiated got closed due to PC rebooted.
  2. Although the directory {ORACLE_BASE} created under the new GRID home, the issue were reoccurring.
Here is the solution:
  1. How to Complete 11gR2 Grid Infrastructure Configuration Assistant(Plug-in) if OUI is not Available (Doc ID 1360798.1)
  2. Ensure the diagnostic_dest is updated on ASM Spfile to the new location before running the rootupgrade.sh


References:
  • Things to Consider Before Upgrading to 11.2.0.3/11.2.0.4 Grid Infrastructure/ASM ( Doc ID 1363369.1) 
  • Things to Consider Before Upgrading to 11.2.0.4 to Avoid Poor Performance or Wrong Results ( Doc ID 1645862.1) 
  • GI rootupgrade.sh on last node: ASM rolling upgrade action failed ( Doc ID 1598959.1) 
  • bug 17449823




Disaster Recovery Simulation test - performed 30 databases failover

Wed, 2014-08-06 05:55
Successfully switched (fail-over) the role of over 30 physical standby databases  this morning as part of the Disaster Recovery (DR) simulation test. Fortunately, there were no technical glitches and hassles during the course of testing as  anticipated. It was indeed a great test and very successful one too.

The next  big challenge to the team would be reconstructing and making in sync those 30 physical standby databases whose range from 100GB to 5TB size.

Anyways, my team loving the challenges and true enjoying every moment.





EID Holidays and things to do

Wed, 2014-07-23 02:07
Looking forward to a much anticipated 9 day EID holiday break to complete the to-do-list which I have been carrying for a while now. Determined to complete some of the writing assignments that I have kept pending for a long period of time now. At the same time, will have to seek the possibilities to exploring the new features of v12.1.0.2 and Exadata as we might we going for the combination in the coming weeks for a Data Warehouse project.

Will surely blog about my test scenarios and will share the inputs on Oracle 12c new features.

I wish everyone a very happy and prosperous EID in advance.

Monthly article publication on Toad World portal

Wed, 2014-06-04 05:50
Let me quickly give you an update if you are wondering why I kept clam, not making much buzz on my blog. Well, over the past one year or so, I have been contributing monthly articles to Toad World news letter, and all my articles are published at their portal. I must encourage you all to visit Toad World website where many Oracle Experts/Gurus contributing/sharing knowledge.

Middle East & North Africa (MENA) 2014 OTN Tour - Tunis/Riyadh/Jeddah/Dubai

Mon, 2014-05-19 04:44
First ever MENA OTN Tour is scheduled From 26-May to 1-June 2014 in Tunis/Saudi Arabia/Dubai major cities.

2 Continents, 3 Countries, 5 Cities:50+ Action-Packed Oracle Sessions.Physical Attendance - Click here to attend this event in-personVirtual Attendance - Click here to virtually attend this event onlineCourtesy of the Oracle Technology Network (OTN) and the ARABOUG, the inaugural 2014 OTN MENA Tour brings a star-studded cast, consisting of some of the world's best Oracle ACEs, ACE Directors and Rock Star Speakers to the region. The tour aims at sharing cutting edge knowledge and independent research in the MENA region, by accomplished Oracle experts from all over the world. This is a 100% FREE event for the benefit of the local Oracle communities.
SpeakersbjoernrostedwardroskeSyedJaffarHussainOsama MustafaMohamedHouriJimCzupyrinskitariqfarooqmikeaultjoelpereshead044InamPic1basheerkhanmohamedcharguiSouhaibAmdouniZarroukHousseMeddingVishnu