RE: ORA-15042: ASM disk "xx" is missing
Date: Wed, 2 Jan 2013 13:12:19 -0500
Message-ID: <CC45762506E4C84A8612307613CDCB2560D33ACF_at_EX7T2-SV08.TDBFG.COM>
No specific view that I am aware of myself my Oracle has a note on how to do this using os tools:
Note id: 452770.1
Use truss on the rbal process when doing the select * from v$asm_disk.
hth
Rui Amaral
Senior Oracle DBA
Oracle Database Engineering, TDBFG
E-Mail: rui.amaral_at_tdsecurities.com| Blackberry: 416-625-3280 |
-----Original Message-----
From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of Fergal Taheny
Sent: Wednesday, January 02, 2013 12:45 PM
To: oracle-l-freelists
Subject: ORA-15042: ASM disk "xx" is missing
Hi,
ASM 11.1.0.7.0
Solaris 10
We have a diskgoup that won't mount. I'm believe it a SAN/disk issue as opposed to an ASM issue but I'm trying to eliminate some guesswork in troubleshooting this.
In the +ASM alert log we have:
NOTE: cache registered group ORADATA number=1 incarn=0x2bc38c91 NOTE: cache began mount (first) of group ORADATA number=1 incarn=0x2bc38c91 Wed Jan 02 14:16:33 2013 NOTE: Assigning number (1,15) to disk (/dev/oradsk/ASMDA7) NOTE: Assigning number (1,10) to disk (/dev/oradsk/ASMDA2) NOTE: Assigning number (1,18) to disk (/dev/oradsk/ASMDA10) NOTE: Assigning number (1,9) to disk (/dev/oradsk/ASMDA1) NOTE: Assigning number (1,19) to disk (/dev/oradsk/ASMDA11) NOTE: Assigning number (1,20) to disk (/dev/oradsk/ASMDA12) NOTE: Assigning number (1,21) to disk (/dev/oradsk/ASMDA13) NOTE: Assigning number (1,22) to disk (/dev/oradsk/ASMDA14) NOTE: Assigning number (1,23) to disk (/dev/oradsk/ASMDA15) NOTE: Assigning number (1,24) to disk (/dev/oradsk/ASMDA16) NOTE: Assigning number (1,25) to disk (/dev/oradsk/ASMDA17) NOTE: Assigning number (1,26) to disk (/dev/oradsk/ASMDA18) NOTE: Assigning number (1,12) to disk (/dev/oradsk/ASMDA4) Wed Jan 02 14:16:37 2013 NOTE: start heartbeating (grp 1)
kfdp_query(): 12
kfdp_queryBg(): 12
NOTE: Assigning number (1,11) to disk () NOTE: Assigning number (1,13) to disk () NOTE: Assigning number (1,14) to disk () NOTE: Assigning number (1,16) to disk () NOTE: Assigning number (1,17) to disk ()kfdp_query(): 13
kfdp_queryBg(): 13
NOTE: cache dismounting group 1/0x2BC38C91 (ORADATA) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0x2BC38C91 (ORADATA) NOTE: cache ending mount (fail) of group ORADATA number=1 incarn=0x2bc38c91kfdp_dismount(): 14
kfdp_dismountBg(): 14
NOTE: De-assigning number (1,9) from disk (/dev/oradsk/ASMDA1) NOTE: De-assigning number (1,10) from disk (/dev/oradsk/ASMDA2) NOTE: De-assigning number (1,12) from disk (/dev/oradsk/ASMDA4) NOTE: De-assigning number (1,15) from disk (/dev/oradsk/ASMDA7) NOTE: De-assigning number (1,18) from disk (/dev/oradsk/ASMDA10) NOTE: De-assigning number (1,19) from disk (/dev/oradsk/ASMDA11) NOTE: De-assigning number (1,20) from disk (/dev/oradsk/ASMDA12) NOTE: De-assigning number (1,21) from disk (/dev/oradsk/ASMDA13) NOTE: De-assigning number (1,22) from disk (/dev/oradsk/ASMDA14) NOTE: De-assigning number (1,23) from disk (/dev/oradsk/ASMDA15) NOTE: De-assigning number (1,24) from disk (/dev/oradsk/ASMDA16) NOTE: De-assigning number (1,25) from disk (/dev/oradsk/ASMDA17) NOTE: De-assigning number (1,26) from disk (/dev/oradsk/ASMDA18)ERROR: diskgroup ORADATA was not mounted
ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15042: ASM disk "17" is missing ORA-15042: ASM disk "16" is missing ORA-15042: ASM disk "14" is missing ORA-15042: ASM disk "13" is missing ORA-15042: ASM disk "11" is missing
ERROR: alter diskgroup oradata mount
So ASM identified 13 disks and couldn't find 5 disks.
The first problem is how to map these missing ASM disks to unix device files.
Looking in /dev/oradsk I find device files for the 13 good disks listed above and also 5 other device files not listed above. I assume these are the device files for the 5 problem disks but I want to prove that.
When I use the od command I can read the file headers for the 13 good disks:
> od -c -N 128 /dev/oradsk/ASMDA1
0000000 \0 202 001 001 \0 \0 \0 \0 200 \0 \0 \t 326 331 223 236 0000020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000040 O R C L D I S K \0 \0 \0 \0 \0 \0 \0 \0 0000060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000100 \n 020 \0 \0 \0 \t 001 003 O R A D A T A _ 0000120 0 0 0 9 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \00000140 \0 \0 \0 \0 \0 \0 \0 \0 O R A D A T A \0 0000160 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0000200
And I get an IO error for the other 5 device files
> od -c -N 128 /dev/oradsk/ASMDA3
od: cannot open ASMDA3: I/O error
Ok still promising but still guessing a bit.
So I look in v$asm_disk and I only see the 13 good disks (same for v$asm_disk_stat and X$KFDSK)
sys_at_+ASM> select path from v$asm_disk;
PATH
/dev/oradsk/ASMDA1
/dev/oradsk/ASMDA2
/dev/oradsk/ASMDA4
/dev/oradsk/ASMDA7
/dev/oradsk/ASMDA10
/dev/oradsk/ASMDA11
/dev/oradsk/ASMDA18
/dev/oradsk/ASMDA13
/dev/oradsk/ASMDA14
/dev/oradsk/ASMDA15
/dev/oradsk/ASMDA16
/dev/oradsk/ASMDA17
/dev/oradsk/ASMDA12
13 rows selected.
Eventually I go back through old alert logs to find the last time the diskgroup was successfully mounted and I find that indeed the five disks I suspected were mounted in the disk group at that time. So that's fairly conclusive but my questions are:
Is there a view in ASM that lists all the disks that belong in a disk group? Including ones asm can't see at present.
Or is this information stored in the file headers? If so how could I read it?
Or does ASM just probe all the disks (as defined in asm_diskstring) to find all the disks in a disk group? And if it does how does it know how many disks to expect? I don't see the "numbers of disks" displayed anywhere in V$ASK_DISKGROUP or in the output of kfed (shown below). Where would I find the number of disks in each disk group?
> kfed read /dev/oradsk/ASMDA1
kfbh.endian: 0 ; 0x000: 0x00 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD kfbh.datfmt: 1 ; 0x003: 0x01 kfbh.block.blk: 0 ; 0x004: T=0 NUMB=0x0 kfbh.block.obj: 2147483657 ; 0x008: TYPE=0x8 NUMB=0x9 kfbh.check: 3604583326 ; 0x00c: 0xd6d9939e kfbh.fcn.base: 0 ; 0x010: 0x00000000 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8 kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000 kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000 kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000 kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000 kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000 kfdhdb.driver.reserved[5]: 0 ; 0x01c: 0x00000000 kfdhdb.compat: 168820736 ; 0x020: 0x0a100000 kfdhdb.dsknum: 9 ; 0x024: 0x0009 kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER kfdhdb.dskname: ORADATA_0009 ; 0x028: length=12 kfdhdb.grpname: ORADATA ; 0x048: length=7 kfdhdb.fgname: ORADATA_0009 ; 0x068: length=12 kfdhdb.capname: ; 0x088: length=0 kfdhdb.crestmp.hi: 32975350 ; 0x0a8: HOUR=0x16 DAYS=0xf MNTH=0xa YEAR=0x7dc kfdhdb.crestmp.lo: 1831326720 ; 0x0ac: USEC=0x0 MSEC=0x1f5 SECS=0x12 MINS=0x1b kfdhdb.mntstmp.hi: 32975859 ; 0x0b0: HOUR=0x13 DAYS=0x1f MNTH=0xa YEAR=0x7dc kfdhdb.mntstmp.lo: 3870520320 ; 0x0b4: USEC=0x0 MSEC=0xdd SECS=0x2b MINS=0x39 kfdhdb.secsize: 512 ; 0x0b8: 0x0200 kfdhdb.blksize: 4096 ; 0x0ba: 0x1000 kfdhdb.ausize: 1048576 ; 0x0bc: 0x00100000 kfdhdb.mfact: 113792 ; 0x0c0: 0x0001bc80 kfdhdb.dsksize: 204760 ; 0x0c4: 0x00031fd8 kfdhdb.pmcnt: 3 ; 0x0c8: 0x00000003 kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001 kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002 kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000 kfdhdb.redomirrors[0]: 0 ; 0x0d8: 0x0000 kfdhdb.redomirrors[1]: 0 ; 0x0da: 0x0000 kfdhdb.redomirrors[2]: 0 ; 0x0dc: 0x0000 kfdhdb.redomirrors[3]: 0 ; 0x0de: 0x0000 kfdhdb.dbcompat: 168820736 ; 0x0e0: 0x0a100000 kfdhdb.grpstmp.hi: 32942731 ; 0x0e4: HOUR=0xb DAYS=0x14 MNTH=0xa YEAR=0x7da kfdhdb.grpstmp.lo: 3678353408 ; 0x0e8: USEC=0x0 MSEC=0x3ce SECS=0x33 MINS=0x36 kfdhdb.ub4spare[0]: 0 ; 0x0ec: 0x00000000 kfdhdb.ub4spare[1]: 0 ; 0x0f0: 0x00000000 kfdhdb.ub4spare[2]: 0 ; 0x0f4: 0x00000000 kfdhdb.ub4spare[3]: 0 ; 0x0f8: 0x00000000 kfdhdb.ub4spare[4]: 0 ; 0x0fc: 0x00000000 kfdhdb.ub4spare[5]: 0 ; 0x100: 0x00000000 kfdhdb.ub4spare[6]: 0 ; 0x104: 0x00000000 kfdhdb.ub4spare[7]: 0 ; 0x108: 0x00000000 kfdhdb.ub4spare[8]: 0 ; 0x10c: 0x00000000 kfdhdb.ub4spare[9]: 0 ; 0x110: 0x00000000 kfdhdb.ub4spare[10]: 0 ; 0x114: 0x00000000 kfdhdb.ub4spare[11]: 0 ; 0x118: 0x00000000 kfdhdb.ub4spare[12]: 0 ; 0x11c: 0x00000000 kfdhdb.ub4spare[13]: 0 ; 0x120: 0x00000000 kfdhdb.ub4spare[14]: 0 ; 0x124: 0x00000000 kfdhdb.ub4spare[15]: 0 ; 0x128: 0x00000000 kfdhdb.ub4spare[16]: 0 ; 0x12c: 0x00000000 kfdhdb.ub4spare[17]: 0 ; 0x130: 0x00000000 kfdhdb.ub4spare[18]: 0 ; 0x134: 0x00000000 kfdhdb.ub4spare[19]: 0 ; 0x138: 0x00000000 kfdhdb.ub4spare[20]: 0 ; 0x13c: 0x00000000 kfdhdb.ub4spare[21]: 0 ; 0x140: 0x00000000 kfdhdb.ub4spare[22]: 0 ; 0x144: 0x00000000 kfdhdb.ub4spare[23]: 0 ; 0x148: 0x00000000 kfdhdb.ub4spare[24]: 0 ; 0x14c: 0x00000000 kfdhdb.ub4spare[25]: 0 ; 0x150: 0x00000000 kfdhdb.ub4spare[26]: 0 ; 0x154: 0x00000000 kfdhdb.ub4spare[27]: 0 ; 0x158: 0x00000000 kfdhdb.ub4spare[28]: 0 ; 0x15c: 0x00000000 kfdhdb.ub4spare[29]: 0 ; 0x160: 0x00000000 kfdhdb.ub4spare[30]: 0 ; 0x164: 0x00000000 kfdhdb.ub4spare[31]: 0 ; 0x168: 0x00000000 kfdhdb.ub4spare[32]: 0 ; 0x16c: 0x00000000 kfdhdb.ub4spare[33]: 0 ; 0x170: 0x00000000 kfdhdb.ub4spare[34]: 0 ; 0x174: 0x00000000 kfdhdb.ub4spare[35]: 0 ; 0x178: 0x00000000 kfdhdb.ub4spare[36]: 0 ; 0x17c: 0x00000000 kfdhdb.ub4spare[37]: 0 ; 0x180: 0x00000000 kfdhdb.ub4spare[38]: 0 ; 0x184: 0x00000000 kfdhdb.ub4spare[39]: 0 ; 0x188: 0x00000000 kfdhdb.ub4spare[40]: 0 ; 0x18c: 0x00000000 kfdhdb.ub4spare[41]: 0 ; 0x190: 0x00000000 kfdhdb.ub4spare[42]: 0 ; 0x194: 0x00000000 kfdhdb.ub4spare[43]: 0 ; 0x198: 0x00000000 kfdhdb.ub4spare[44]: 0 ; 0x19c: 0x00000000 kfdhdb.ub4spare[45]: 0 ; 0x1a0: 0x00000000 kfdhdb.ub4spare[46]: 0 ; 0x1a4: 0x00000000 kfdhdb.ub4spare[47]: 0 ; 0x1a8: 0x00000000 kfdhdb.ub4spare[48]: 0 ; 0x1ac: 0x00000000 kfdhdb.ub4spare[49]: 0 ; 0x1b0: 0x00000000 kfdhdb.ub4spare[50]: 0 ; 0x1b4: 0x00000000 kfdhdb.ub4spare[51]: 0 ; 0x1b8: 0x00000000 kfdhdb.ub4spare[52]: 0 ; 0x1bc: 0x00000000 kfdhdb.ub4spare[53]: 0 ; 0x1c0: 0x00000000 kfdhdb.ub4spare[54]: 0 ; 0x1c4: 0x00000000 kfdhdb.ub4spare[55]: 0 ; 0x1c8: 0x00000000 kfdhdb.ub4spare[56]: 0 ; 0x1cc: 0x00000000 kfdhdb.ub4spare[57]: 0 ; 0x1d0: 0x00000000 kfdhdb.acdb.aba.seq: 0 ; 0x1d4: 0x00000000 kfdhdb.acdb.aba.blk: 0 ; 0x1d8: 0x00000000 kfdhdb.acdb.ents: 0 ; 0x1dc: 0x0000 kfdhdb.acdb.ub2spare: 0 ; 0x1de: 0x0000
Thanks,
Fergal
-- http://www.freelists.org/webpage/oracle-l NOTICE: Confidential message which may be privileged. Unauthorized use/disclosure prohibited. If received in error, please go to www.td.com/legal for instructions. AVIS : Message confidentiel dont le contenu peut être privilégié. Utilisation/divulgation interdites sans permission. Si reçu par erreur, prière d'aller au www.td.com/francais/avis_juridique pour des instructions. -- http://www.freelists.org/webpage/oracle-lReceived on Wed Jan 02 2013 - 19:12:19 CET