Re: ASM disk path in exadata

From: Andy Colvin <acolvin_at_enkitec.com>
Date: Tue, 21 Feb 2012 07:46:43 -0600
Message-Id: <007091A2-C6EC-4024-B712-2EC0EDCEB77B_at_enkitec.com>

Right�the ASM disks on Exadata do not exist on the database servers as physical devices. It's not like a typical system that uses block or raw devices for the ASM disks. They're connected using the iDB protocol over RDS or UDP. [enkdb01:root] /root
> fdisk -l

Disk /dev/sda: 291.9 GB, 291999055872 bytes 255 heads, 63 sectors/track, 35500 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1        7832    62910508+  83  Linux
/dev/sda2            7833        9920    16771860   82  Linux swap / Solaris
/dev/sda3            9921       35500   205471350   83  Linux

[enkdb01:root] /root
> kfod disks=all

 Disk          Size Path                                     User     Group   
================================================================================
   1:    1501184 Mb o/192.168.12.3/DATA_CD_00_enkcel01       <unknown> <unknown>
   2:    1501184 Mb o/192.168.12.3/DATA_CD_01_enkcel01       <unknown> <unknown>
   3:    1501184 Mb o/192.168.12.3/DATA_CD_02_enkcel01       <unknown> <unknown>
   4:    1501184 Mb o/192.168.12.3/DATA_CD_03_enkcel01       <unknown> <unknown>
   5:    1501184 Mb o/192.168.12.3/DATA_CD_04_enkcel01       <unknown> <unknown>
   6:    1501184 Mb o/192.168.12.3/DATA_CD_05_enkcel01       <unknown> <unknown>
   7:    1501184 Mb o/192.168.12.3/DATA_CD_06_enkcel01       <unknown> <unknown>
   8:    1501184 Mb o/192.168.12.3/DATA_CD_07_enkcel01       <unknown> <unknown>
   9:    1501184 Mb o/192.168.12.3/DATA_CD_08_enkcel01       <unknown> <unknown>
  10:    1501184 Mb o/192.168.12.3/DATA_CD_09_enkcel01       <unknown> <unknown>
  11:    1501184 Mb o/192.168.12.3/DATA_CD_10_enkcel01       <unknown> <unknown>
  12:    1501184 Mb o/192.168.12.3/DATA_CD_11_enkcel01       <unknown> <unknown>
  13:      29808 Mb o/192.168.12.3/DBFS_DG_CD_02_enkcel01    <unknown> <unknown>
  14:      29808 Mb o/192.168.12.3/DBFS_DG_CD_03_enkcel01    <unknown> <unknown>
  15:      29808 Mb o/192.168.12.3/DBFS_DG_CD_04_enkcel01    <unknown> <unknown>
  16:      29808 Mb o/192.168.12.3/DBFS_DG_CD_05_enkcel01    <unknown> <unknown>
  17:      29808 Mb o/192.168.12.3/DBFS_DG_CD_06_enkcel01    <unknown> <unknown>
  18:      29808 Mb o/192.168.12.3/DBFS_DG_CD_07_enkcel01    <unknown> <unknown>
  19:      29808 Mb o/192.168.12.3/DBFS_DG_CD_08_enkcel01    <unknown> <unknown>
  20:      29808 Mb o/192.168.12.3/DBFS_DG_CD_09_enkcel01    <unknown> <unknown>
  21:      29808 Mb o/192.168.12.3/DBFS_DG_CD_10_enkcel01    <unknown> <unknown>
  22:      29808 Mb o/192.168.12.3/DBFS_DG_CD_11_enkcel01    <unknown> <unknown>
  23:     375344 Mb o/192.168.12.3/RECO_CD_00_enkcel01       <unknown> <unknown>
  24:     375344 Mb o/192.168.12.3/RECO_CD_01_enkcel01       <unknown> <unknown>
  25:     375344 Mb o/192.168.12.3/RECO_CD_02_enkcel01       <unknown> <unknown>
  26:     375344 Mb o/192.168.12.3/RECO_CD_03_enkcel01       <unknown> <unknown>
  27:     375344 Mb o/192.168.12.3/RECO_CD_04_enkcel01       <unknown> <unknown>
  28:     375344 Mb o/192.168.12.3/RECO_CD_05_enkcel01       <unknown> <unknown>
  29:     375344 Mb o/192.168.12.3/RECO_CD_06_enkcel01       <unknown> <unknown>
  30:     375344 Mb o/192.168.12.3/RECO_CD_07_enkcel01       <unknown> <unknown>
  31:     375344 Mb o/192.168.12.3/RECO_CD_08_enkcel01       <unknown> <unknown>
  32:     375344 Mb o/192.168.12.3/RECO_CD_09_enkcel01       <unknown> <unknown>
  33:     375344 Mb o/192.168.12.3/RECO_CD_10_enkcel01       <unknown> <unknown>
  34:     375344 Mb o/192.168.12.3/RECO_CD_11_enkcel01       <unknown> <unknown>
  35:    1501184 Mb o/192.168.12.4/DATA_CD_00_enkcel02       <unknown> <unknown>
  36:    1501184 Mb o/192.168.12.4/DATA_CD_01_enkcel02       <unknown> <unknown>
  37:    1501184 Mb o/192.168.12.4/DATA_CD_02_enkcel02       <unknown> <unknown>
  38:    1501184 Mb o/192.168.12.4/DATA_CD_03_enkcel02       <unknown> <unknown>
  39:    1501184 Mb o/192.168.12.4/DATA_CD_04_enkcel02       <unknown> <unknown>
  40:    1501184 Mb o/192.168.12.4/DATA_CD_05_enkcel02       <unknown> <unknown>
  41:    1501184 Mb o/192.168.12.4/DATA_CD_06_enkcel02       <unknown> <unknown>
  42:    1501184 Mb o/192.168.12.4/DATA_CD_07_enkcel02       <unknown> <unknown>
  43:    1501184 Mb o/192.168.12.4/DATA_CD_08_enkcel02       <unknown> <unknown>
  44:    1501184 Mb o/192.168.12.4/DATA_CD_09_enkcel02       <unknown> <unknown>
  45:    1501184 Mb o/192.168.12.4/DATA_CD_10_enkcel02       <unknown> <unknown>
  46:    1501184 Mb o/192.168.12.4/DATA_CD_11_enkcel02       <unknown> <unknown>
  47:      29808 Mb o/192.168.12.4/DBFS_DG_CD_02_enkcel02    <unknown> <unknown>
  48:      29808 Mb o/192.168.12.4/DBFS_DG_CD_03_enkcel02    <unknown> <unknown>
  49:      29808 Mb o/192.168.12.4/DBFS_DG_CD_04_enkcel02    <unknown> <unknown>
  50:      29808 Mb o/192.168.12.4/DBFS_DG_CD_05_enkcel02    <unknown> <unknown>
  51:      29808 Mb o/192.168.12.4/DBFS_DG_CD_06_enkcel02    <unknown> <unknown>
  52:      29808 Mb o/192.168.12.4/DBFS_DG_CD_07_enkcel02    <unknown> <unknown>
  53:      29808 Mb o/192.168.12.4/DBFS_DG_CD_08_enkcel02    <unknown> <unknown>
  54:      29808 Mb o/192.168.12.4/DBFS_DG_CD_09_enkcel02    <unknown> <unknown>
  55:      29808 Mb o/192.168.12.4/DBFS_DG_CD_10_enkcel02    <unknown> <unknown>
  56:      29808 Mb o/192.168.12.4/DBFS_DG_CD_11_enkcel02    <unknown> <unknown>
  57:     375344 Mb o/192.168.12.4/RECO_CD_00_enkcel02       <unknown> <unknown>
  58:     375344 Mb o/192.168.12.4/RECO_CD_01_enkcel02       <unknown> <unknown>
  59:     375344 Mb o/192.168.12.4/RECO_CD_02_enkcel02       <unknown> <unknown>
  60:     375344 Mb o/192.168.12.4/RECO_CD_03_enkcel02       <unknown> <unknown>
  61:     375344 Mb o/192.168.12.4/RECO_CD_04_enkcel02       <unknown> <unknown>
  62:     375344 Mb o/192.168.12.4/RECO_CD_05_enkcel02       <unknown> <unknown>
  63:     375344 Mb o/192.168.12.4/RECO_CD_06_enkcel02       <unknown> <unknown>
  64:     375344 Mb o/192.168.12.4/RECO_CD_07_enkcel02       <unknown> <unknown>
  65:     375344 Mb o/192.168.12.4/RECO_CD_08_enkcel02       <unknown> <unknown>
  66:     375344 Mb o/192.168.12.4/RECO_CD_09_enkcel02       <unknown> <unknown>
  67:     375344 Mb o/192.168.12.4/RECO_CD_10_enkcel02       <unknown> <unknown>
  68:     375344 Mb o/192.168.12.4/RECO_CD_11_enkcel02       <unknown> <unknown>
  69:    1501184 Mb o/192.168.12.5/DATA_CD_00_enkcel03       <unknown> <unknown>
  70:    1501184 Mb o/192.168.12.5/DATA_CD_01_enkcel03       <unknown> <unknown>
  71:    1501184 Mb o/192.168.12.5/DATA_CD_02_enkcel03       <unknown> <unknown>
  72:    1501184 Mb o/192.168.12.5/DATA_CD_03_enkcel03       <unknown> <unknown>
  73:    1501184 Mb o/192.168.12.5/DATA_CD_04_enkcel03       <unknown> <unknown>
  74:    1501184 Mb o/192.168.12.5/DATA_CD_05_enkcel03       <unknown> <unknown>
  75:    1501184 Mb o/192.168.12.5/DATA_CD_06_enkcel03       <unknown> <unknown>
  76:    1501184 Mb o/192.168.12.5/DATA_CD_07_enkcel03       <unknown> <unknown>
  77:    1501184 Mb o/192.168.12.5/DATA_CD_08_enkcel03       <unknown> <unknown>
  78:    1501184 Mb o/192.168.12.5/DATA_CD_09_enkcel03       <unknown> <unknown>
  79:    1501184 Mb o/192.168.12.5/DATA_CD_10_enkcel03       <unknown> <unknown>
  80:    1501184 Mb o/192.168.12.5/DATA_CD_11_enkcel03       <unknown> <unknown>
  81:      29808 Mb o/192.168.12.5/DBFS_DG_CD_02_enkcel03    <unknown> <unknown>
  82:      29808 Mb o/192.168.12.5/DBFS_DG_CD_03_enkcel03    <unknown> <unknown>
  83:      29808 Mb o/192.168.12.5/DBFS_DG_CD_04_enkcel03    <unknown> <unknown>
  84:      29808 Mb o/192.168.12.5/DBFS_DG_CD_05_enkcel03    <unknown> <unknown>
  85:      29808 Mb o/192.168.12.5/DBFS_DG_CD_06_enkcel03    <unknown> <unknown>
  86:      29808 Mb o/192.168.12.5/DBFS_DG_CD_07_enkcel03    <unknown> <unknown>
  87:      29808 Mb o/192.168.12.5/DBFS_DG_CD_08_enkcel03    <unknown> <unknown>
  88:      29808 Mb o/192.168.12.5/DBFS_DG_CD_09_enkcel03    <unknown> <unknown>
  89:      29808 Mb o/192.168.12.5/DBFS_DG_CD_10_enkcel03    <unknown> <unknown>
  90:      29808 Mb o/192.168.12.5/DBFS_DG_CD_11_enkcel03    <unknown> <unknown>
  91:     375344 Mb o/192.168.12.5/RECO_CD_00_enkcel03       <unknown> <unknown>
  92:     375344 Mb o/192.168.12.5/RECO_CD_01_enkcel03       <unknown> <unknown>
  93:     375344 Mb o/192.168.12.5/RECO_CD_02_enkcel03       <unknown> <unknown>
  94:     375344 Mb o/192.168.12.5/RECO_CD_03_enkcel03       <unknown> <unknown>
  95:     375344 Mb o/192.168.12.5/RECO_CD_04_enkcel03       <unknown> <unknown>
  96:     375344 Mb o/192.168.12.5/RECO_CD_05_enkcel03       <unknown> <unknown>
  97:     375344 Mb o/192.168.12.5/RECO_CD_06_enkcel03       <unknown> <unknown>
  98:     375344 Mb o/192.168.12.5/RECO_CD_07_enkcel03       <unknown> <unknown>
  99:     375344 Mb o/192.168.12.5/RECO_CD_08_enkcel03       <unknown> <unknown>
 100:     375344 Mb o/192.168.12.5/RECO_CD_09_enkcel03       <unknown> <unknown>
 101:     375344 Mb o/192.168.12.5/RECO_CD_10_enkcel03       <unknown> <unknown>
 102:     375344 Mb o/192.168.12.5/RECO_CD_11_enkcel03       <unknown> <unknown>
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME                                                          
================================================================================
     +ASM1 /u01/app/11.2.0.3/grid                                               
     +ASM2 /u01/app/11.2.0.3/grid

Andy Colvin

Principal Consultant
Enkitec
andy.colvin_at_enkitec.com
http://blog.oracle-ninja.com

On Feb 21, 2012, at 6:07 AM, Frits Hoogland wrote:

> Content-Transfer-Encoding: quoted-printable
> Content-Type: text/plain;
> charset=us-ascii
>
> There is no udev/path translation on the database/computing layer in > exadata.
>
> During startup, ASM first needs to find the disks. It does that by > picking up the cellserver information (addresses) in > /etc/oracle/cell/network-config/cellip.ora
> Because the discovery string is set to "o/*/*", it picks up all > griddisks available on the cellservers defined in cellip.ora
>
> Because of the entirely different path, ASM knows it needs to > communicate over the network to get to the griddisks, instead of using a > locally attached disk. This combination is RDS (infiniband) on Exadata, > but can be UDP (ethernet).
>
> In fact, the discovery can manually be done by using kfod:
> kfod asm_diskstring="o/*/*", disks=all
> this is what the asmca uses behind the GUI to discover disks, also on > normal platforms.
> When specifying the o/*/* diskstring, kfod too reads > /etc/oracle/cell/network-config/cellip.ora to know where to look.
>
>
> Frits Hoogland
>
> http://fritshoogland.wordpress.com
> mailto:frits.hoogland_at_gmail.com
> cell: +31 6 53569942
>
> On Feb 21, 2012, at 10:32 AM, gaurav mehta wrote:
>

>> I have been trying to figure out how exadata griddisks are presented > as candidate asm disks.  We know that the disk discovery path for asm > instances using exadata is "o/<cell_ip_address>*/<griddisk_name>*".  I > was wondering what actual path / udev in the operating system on the db > machine does this translate to ?.   
>> --
>> http://www.freelists.org/webpage/oracle-l
>>  
>>

>
>
>
>
> --
> http://www.freelists.org/webpage/oracle-l
>
>

--
http://www.freelists.org/webpage/oracle-l

Received on Tue Feb 21 2012 - 07:46:43 CST

This message: [ Message body ]
Next message: Orastaff: "* Oracle DBA Needed in Upstate New York-- Great Company!"
Previous message: rjamya: "Re: Question about: DBMS_AQ.DEQUEUE"
In reply to: Frits Hoogland: "Re: ASM disk path in exadata"
Next in thread: Gaurav Mehta: "Re: ASM disk path in exadata"
Reply: Gaurav Mehta: "Re: ASM disk path in exadata"

Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ]

Original text of this message