RE: Anyone running a 10.2.0.3 RAC on Linux?
Date: Thu, 28 Feb 2013 17:45:19 +0000
Message-ID: <9F15274DDC89C24387BE933E68BE3FD30827A098_at_MISOUT7MSGUSR9D.ITServices.sbc.com>
We had tons of RAC on 10.2 on Red Hat Linux. You will find that 10.2.0.4 and 10.2.0.5 are far more stable. 11.2.0.3 is very reliable. I would suggest you make plans to go to 10.2.0.5 in the near term, and plan on 11.2.0.3 long term. Also pay attention to things like rpms for glibc, gcc etc: sometimes sysadmins patch those and forget to tell the DBA, so the binaries are never relinked. Same for kernel versions. This can cause subtle problems and stability issues. You can also use things like RACCHECK from Oracle and the Cluster Verification Utility- cvu is good for the rpm checking.
-----Original Message-----
From: CRISLER, JON A
Sent: Thursday, February 28, 2013 12:26 PM
To: 'oratune_at_yahoo.com'; Christopher.Taylor2_at_parallon.net; Dave.Herring_at_acxiom.com; oracle-l_at_freelists.org
Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
This is something we do for 11.1 and 11.2 RAC, but I don't recall ever putting it on a 10.2 db. SQL> alter system set "_high_priority_processes"='LMS*|VKTM|LGWR' scope=spfile sid='*';
System altered.
I don't think the VKTM process exists outside of 11.x RAC (it's a kernel time monitor IIRC much like NTP but specific to RAC). Putting LGWR was a response to a heavily loaded system where LGWR was sometimes starved for resources, and caused excessive waits as other nodes were waiting on the dirty bock to be written out.
The parameter _high_priority_processes at least in linux tells the OS to make it a RT (real time) process.
You can do the same thing in the operating system as well using the chrt (Linux) command. Keep in mind that this is not so much changing priorities and it is changing it to a RT process- much more aggressive change :)
chrt -p (pid) display RT attribs chrt -p 1 (pid) change to RT chrt -o -p (pid) change back to non-RT
Fooling around in this area could be dangerous, so be careful and test it on a non-prod system with a realistic workload.
-----Original Message-----
From: oracle-l-bounce_at_freelists.org [mailto:oracle-l-bounce_at_freelists.org] On Behalf Of David Fitzjarrell
Sent: Wednesday, February 27, 2013 4:22 PM
To: Christopher.Taylor2_at_parallon.net; Dave.Herring_at_acxiom.com; oracle-l_at_freelists.org
Subject: Re: Anyone running a 10.2.0.3 RAC on Linux?
My guess would be the latter case -- new 'features' and parameters rarely work as expected or intended when introduced. As stated before 10.2.0.3 was a rough release on many fronts.
David Fitzjarrell
From: "Christopher.Taylor2_at_parallon.net" <Christopher.Taylor2_at_parallon.net> To: Dave.Herring_at_acxiom.com; oratune_at_yahoo.com; oracle-l_at_freelists.org Sent: Wednesday, February 27, 2013 1:51 PM Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
My fault - read your version list too fast. I'm betting that parameter is too aggressive in 10.2.0.3 - or doesn't work as expected. Thanks though!
Chris
From: Herring Dave - dherri [mailto:Dave.Herring_at_acxiom.com] Sent: Wednesday, February 27, 2013 2:50 PM To: Taylor Christopher - Nashville; oratune_at_yahoo.com; oracle-l_at_freelists.org Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
No 10.2.0.3, just 10.2.0.2, 10.2.0.4, 11.2.0.1, and 11.2.0.3. It looks this is a new param added in 10.2.0.3.
10.2.0.2
No rows.
10.2.0.4
NAME VALUE ISDEFAULT ISMODIFIED
------------------------- ---------- --------- --------------------------------
_high_priority_processes LMS* TRUE FALSE
11.2.0.1
NAME VALUE ISDEFAULT ISMODIFIED
------------------------- ---------- --------- ----------
_high_priority_processes LMS*|VKTM TRUE FALSE
11.2.0.3
NAME VALUE ISDEFAULT ISMODIFIED
------------------------- ---------- --------- ----------
_high_priority_processes LMS*|VKTM TRUE FALSE
DAVID HERRING
DBA
Acxiom Corporation
EML dave.herring_at_acxiom.com<mailto:dave.herring_at_acxiom.com> TEL 630.944.4762 MBL 630.430.5988
1501 Opus Pl, Downers Grove, IL 60515, USA WWW.ACXIOM.COM<http://www.acxiom.com/>
[cid:image001.png_at_01CE14F9.E81D9C70]<http://www.facebook.com/acxiomcor> [cid:image002.png_at_01CE14F9.E81D9C70] <http://www.linkedin.com/groupRegistration?gid)0173> [cid:image003.png@01CE14F9.E81D9C70] <http://twitter.com/acxio>
[cid:image004.png_at_01CE14F9.E81D9C70]
From: Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net> [mailto:Christopher.Taylor2_at_parallon.net] Sent: Wednesday, February 27, 2013 2:37 PM To: Herring Dave - dherri; oratune_at_yahoo.com<mailto:oratune_at_yahoo.com>; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org> Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
Ran SQL to list hidden parameters:
NAME VALUE ISDEFAULT ISMODIFIED
-------------------------------------------------- -------------------------------------------------- --------- --------------------------------
_high_priority_processes LMS* TRUE FALSE
David H - curious if you can run this as sysdba and see if its the same in your 10.2.0.3:
SELECT name,value,ISDEFAULT,ISMODIFIED FROM (SELECT x.inst_id AS INST_ID,
x.indx + 1 AS NUM, ksppinm AS NAME, ksppity AS TYPE, ksppstvl AS VALUE, ksppstdvl AS DISPLAY_VALUE, ksppstdf AS ISDEFAULT, DECODE (BITAND (ksppiflg / 256, 1), 1, 'TRUE', 'FALSE') AS ISSES_MODIFIABLE, DECODE (BITAND (ksppiflg / 65536, 3), 1, 'IMMEDIATE', 2, 'DEFERRED', 3, 'IMMEDIATE', 'FALSE') AS ISSYS_MODIFIABLE, DECODE (BITAND (ksppiflg, 4), 4, 'FALSE', DECODE (BITAND (ksppiflg / 65536, 3), 0, 'FALSE', 'TRUE')) AS ISINSTANCE_MODIFIABLE, DECODE (BITAND (ksppstvf, 7), 1, 'MODIFIED', 4, 'SYSTEM_MOD', 'FALSE') AS ISMODIFIED, DECODE (BITAND (ksppstvf, 2), 2, 'TRUE', 'FALSE') AS ISADJUSTED, DECODE (BITAND (ksppilrmflg / 64, 1), 1, 'TRUE', 'FALSE') AS ISDEPRECATED, ksppdesc AS DESCRIPTION, ksppstcmnt AS UPDATE_COMMENT, ksppihash AS HASH FROM sys.x$ksppi x, sys.x$ksppcv y WHERE (x.indx = y.indx)
)
where NAME like lower('%&ENTER_PARM_NAME%') /
From: Herring Dave - dherri [mailto:Dave.Herring_at_acxiom.com] Sent: Wednesday, February 27, 2013 2:26 PM To: David Fitzjarrell; Taylor Christopher - Nashville; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org> Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
I checked 10.2.0.2, 10.2.0.4, 11.2.0.1, and 11.2.0.3 and all are at "-2". All clusters are on RHEL.
DAVID HERRING
DBA
Acxiom Corporation
EML dave.herring_at_acxiom.com<mailto:dave.herring_at_acxiom.com> TEL 630.944.4762 MBL 630.430.5988
1501 Opus Pl, Downers Grove, IL 60515, USA WWW.ACXIOM.COM<http://www.acxiom.com/>
[cid:image001.png_at_01CE14F9.E81D9C70]<http://www.facebook.com/acxiomcor> [cid:image002.png_at_01CE14F9.E81D9C70] <http://www.linkedin.com/groupRegistration?gid)0173> [cid:image003.png@01CE14F9.E81D9C70] <http://twitter.com/acxio>
[cid:image004.png_at_01CE14F9.E81D9C70]
From: David Fitzjarrell [mailto:oratune_at_yahoo.com] Sent: Wednesday, February 27, 2013 2:23 PM To: Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>; Herring Dave - dherri; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org> Subject: Re: Anyone running a 10.2.0.3 RAC on Linux?
Yes, it does. My former coworker reports that on their lone 10.2.0.3 RAC on linux the lms processes are running at a priority of 90, which also seems odd as Dave Herring reports the same priority on his 10.2.0.2 RAC databases as I do for our 11.2.0.3 databases (-2). It would appear that 10.2.0.3 is doing some 'interesting' things with some of the cluster services. Then, 10.2.0.3 wasn't very good on statistics, either, causing other performance problems.
David Fitzjarrell
From: "Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>" <Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>> To: Dave.Herring_at_acxiom.com<mailto:Dave.Herring_at_acxiom.com>; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org> Sent: Wednesday, February 27, 2013 7:45 AM Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
So does -51 PRIORITY seem crazy to you as it does to me for LMS processes?
Chris
-----Original Message-----
From: Herring Dave - dherri [mailto:Dave.Herring_at_acxiom.com<mailto:Dave.Herring_at_acxiom.com>]
Sent: Wednesday, February 27, 2013 8:44 AM
To: Taylor Christopher - Nashville; oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org>
Subject: RE: Anyone running a 10.2.0.3 RAC on Linux?
Chris,
What flavor and rev of Linux? DRM is pretty buggy under 10g and because of crashes related to it we've done a bit of research into it. The best thing we found was to target longer running cursors to avoid remastering in the first place. I know that sounds like a no-brainer - tune long running cursors - but in this case it was tuning cursors involving data more likely to be shared across the nodes and of those cursors, hit the longest running ones first.
I've got a number of clusters on RHEL 4.6. They're all 10.2.0.2:
% for X in 0 1 2 3; do ssh oracle_at_xxx${X} 'ps -u oracle -O pid,user,priority,nice,vsize,rss,share,pcpu,size | grep lms | grep -v grep'; done 18256 18256 oracle -2 - 151180 224496 - 1.9 2152 S ? 18:47:39 asm_lms0_+ASM1 19345 19345 oracle -2 - 161024 17858588 - 4.3 2744 S ? 1-17:09:49 ora_lms0_XXXXXX1 19349 19349 oracle -2 - 152704 17859424 - 3.0 2616 S ? 1-04:43:51 ora_lms1_XXXXXX1 19353 19353 oracle -2 - 152704 17859644 - 3.2 2616 S ? 1-06:23:23 ora_lms2_XXXXXX1 19357 19357 oracle -2 - 152704 17854440 - 3.0 2616 S ? 1-04:58:33 ora_lms3_XXXXXX1 19361 19361 oracle -2 - 152704 17857896 - 3.1 2616 S ? 1-06:08:38 ora_lms4_XXXXXX1 19365 19365 oracle -2 - 152704 17859636 - 3.0 2616 S ? 1-04:36:10 ora_lms5_XXXXXX1 974 974 oracle -2 - 152704 17859744 - 4.4 2616 R ? 1-17:51:04 ora_lms0_XXXXXX2 982 982 oracle -2 - 151660 17857316 - 3.1 2616 S ? 1-05:42:41 ora_lms1_XXXXXX2 990 990 oracle -2 - 152704 17859196 - 3.3 2616 R ? 1-07:18:02 ora_lms2_XXXXXX2 996 996 oracle -2 - 151660 17857312 - 3.1 2616 S ? 1-05:41:33 ora_lms3_XXXXXX21004 1004 oracle -2 - 152704 17859388 - 3.2 2616 S ? 1-06:25:44 ora_lms4_XXXXXX2 1012 1012 oracle -2 - 151660 17857436 - 3.1 2616 S ? 1-05:57:22 ora_lms5_XXXXXX2
32754 32754 oracle -2 - 151180 224500 - 2.0 2152 S ? 18:59:35 asm_lms0_+ASM2 2876 2876 oracle -2 - 152704 17855760 - 3.8 2616 S ? 22:23:14 ora_lms0_XXXXXX3 2880 2880 oracle -2 - 152704 17855616 - 2.4 2616 S ? 14:07:36 ora_lms1_XXXXXX3 2893 2893 oracle -2 - 152704 17855504 - 2.5 2616 S ? 14:52:41 ora_lms2_XXXXXX3 2914 2914 oracle -2 - 151660 17853452 - 2.4 2616 S ? 14:19:18 ora_lms3_XXXXXX3 2918 2918 oracle -2 - 151664 17853728 - 2.4 2620 S ? 14:34:36 ora_lms4_XXXXXX3 2924 2924 oracle -2 - 151664 17853452 - 2.4 2620 S ? 14:25:32 ora_lms5_XXXXXX3 22555 22555 oracle -2 - 151180 221224 - 1.7 2152 R ? 10:13:15 asm_lms0_+ASM3 9304 9304 oracle -2 - 151180 210960 - 2.0 2152 S ? 18:59:35 asm_lms0_+ASM4 9892 9892 oracle -2 - 151792 17853164 - 4.4 2748 S ? 1-18:08:04 ora_lms0_XXXXXX4 9909 9909 oracle -2 - 161028 17853108 - 3.1 2748 R ? 1-06:06:52 ora_lms1_XXXXXX4 9933 9933 oracle -2 - 151792 17852424 - 3.3 2748 S ? 1-07:40:15 ora_lms2_XXXXXX4 9937 9937 oracle -2 - 151792 17852528 - 3.1 2748 S ? 1-06:04:05 ora_lms3_XXXXXX49941 9941 oracle -2 - 151792 17852492 - 3.3 2748 S ? 1-07:27:18 ora_lms4_XXXXXX4 9945 9945 oracle -2 - 151792 17852584 - 3.2 2748 S ? 1-06:18:26 ora_lms5_XXXXXX4
DAVID HERRING
DBA
Acxiom Corporation
EML dave.herring_at_acxiom.com<mailto:dave.herring_at_acxiom.com> TEL 630.944.4762 MBL 630.430.5988
1501 Opus Pl, Downers Grove, IL 60515, USA http://www.acxiom.com/
-----Original Message-----
From: oracle-l-bounce_at_freelists.org<mailto:oracle-l-bounce_at_freelists.org> [mailto:oracle-l-bounce_at_freelists.org<mailto:oracle-l-bounce_at_freelists.org>] On Behalf Of Christopher.Taylor2_at_parallon.net<mailto:Christopher.Taylor2_at_parallon.net>
Sent: Tuesday, February 26, 2013 6:39 PM
To: oracle-l_at_freelists.org<mailto:oracle-l_at_freelists.org>
Subject: Anyone running a 10.2.0.3 RAC on Linux?
I'm curious what priority your LMS processes run in? One of our DBAs here has a system that is giving him fits and I was looking at it and it seems [to me] that LMS priority for his clusters is way too low but I'm still fairly new to RAC and digging into the internals so wanted to get your thoughts?
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
...
...
9022 oracle -51 0 12.2g 11g 11g S 0.3 17.6 1636:16 ora_lms0_XXXXXXXP6N1
9026 oracle -51 0 12.2g 11g 11g S 0.3 17.6 1252:31 ora_lms1_XXXXXXXP6N1
16296 oracle -51 0 8352m 2.5g 2.5g S 2.3 3.9 7149:56 ora_lms0_XXXXXXXP1N1 16314 oracle -51 0 8352m 2.5g 2.5g S 2.3 3.9 6761:01 ora_lms1_XXXXXXXP1N1 18902 oracle -51 0 8344m 2.2g 2.2g S 1.0 3.5 1822:21 ora_lms0_XXXXXXXP2N1 18910 oracle -51 0 8344m 2.2g 2.2g S 1.0 3.5 1611:40 ora_lms1_XXXXXXXP2N1 21547 oracle -51 0 10.1g 3.1g 3.1g S 2.3 5.0 2742:04 ora_lms0_XXXXXXXP3N1 21551 oracle -51 0 10.1g 3.1g 3.1g S 2.6 5.0 2444:59 ora_lms1_XXXXXXXP3N1 24777 oracle -51 0 8344m 1.6g 1.5g S 1.0 2.5 2769:58 ora_lms0_XXXXXXXP4N1 24781 oracle -51 0 8344m 1.6g 1.5g S 1.0 2.5 2596:55 ora_lms1_XXXXXXXP4N1 29630 oracle -51 0 275m 69m 55m S 0.0 0.1 845:01.75 asm_lms0_+ASM1 30825 oracle -51 0 8344m 2.0g 1.9g S 0.0 3.1 1656:07 ora_lms0_XXXXXXXP5N1 30832 oracle -51 0 8344m 2.0g 1.9g S 0.3 3.1 1417:34 ora_lms1_XXXXXXXP5N1
Chris
-- http://www.freelists.org/webpage/oracle-l *************************************************************************** The information contained in this communication is confidential, is intended only for the use of the recipient named above, and may be legally privileged. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please resend this communication to the sender and delete the original message or any copy of it from your computer system. Thank You. **************************************************************************** -- http://www.freelists.org/webpage/oracle-l -- http://www.freelists.org/webpage/oracle-l -- http://www.freelists.org/webpage/oracle-l -- http://www.freelists.org/webpage/oracle-lReceived on Thu Feb 28 2013 - 18:45:19 CET