Oracle FAQ Your Portal to the Oracle Knowledge Grid
HOME | ASK QUESTION | ADD INFO | SEARCH | E-MAIL US
 

Home -> Community -> Mailing Lists -> Oracle-L -> RE: OT : kernel using 75% of CPU

RE: OT : kernel using 75% of CPU

From: Kevin Lange <kgel_at_ppoone.com>
Date: Tue, 28 Aug 2001 14:04:22 -0700
Message-ID: <F001.0037A6FC.20010828142041@fatcity.com> 







<FONT face=Arial color=#0000ff 


size=2>Jerry;


  


Has there been any system parameter changes lately ??
<FONT face=Arial color=#0000ff 


size=2> 


I 


don't know about on your system, but on our AIX box there was a parameter called 
MBUFs that dealt with Communication Buffers.  Now, you would not think this 
would have any consideration on the database, but it did.
<FONT face=Arial color=#0000ff 


size=2> 


MBUFS 



is the Maximum Allowable Communication Buffer on an AIX system.   We 
thought that it dealt with the networking and set it up to its maximum of 64 
Meg.   We also did not think this would bother Oracle.... boy were we 
wrong.


<FONT face=Arial color=#0000ff 


size=2> 


<FONT face=Arial color=#0000ff 


size=2>Apparently, its also Interprocess comminications as well.  And, 
since Oracle is greedy, it likes to take all the memory the system will give 
it.  It turned out that each process that was started (i.e. every user who 
logged on ) grabbed the maximum memory setup by MBUFS.  So, take a 900 Meg 
SGA and add onto it a 1 Meg Sort area per user and a 64 Meg MBUF per user and 
350 users and you can see why our 4GB of memory went real 
fast.


<FONT face=Arial color=#0000ff 


size=2> 


<FONT face=Arial color=#0000ff 


size=2>Sometimes, things that don't appear to be related can all of a sudden 
jump up and bite you.


<FONT face=Arial color=#0000ff 


size=2> 


<FONT face=Arial color=#0000ff 


size=2>Kevin



  <FONT face=Tahoma 


  size=2>-----Original Message-----From: Jerry C 
  [mailto:usidba_at_yahoo.com]Sent: Tuesday, August 28, 2001 4:46 
  PMTo: Multiple recipients of list ORACLE-LSubject: Re: 
  OT : kernel using 75% of CPU


  Guy,


   



  Thank you very very much, this is a great 
  explanation, and is much appreciated.


   



  To answer some of your questions (and add 
  a few!):


   



  Yes, our client is experiencing 


  performance problems.


   



  vmstat and swap -s seem to show some 


  swapping:


   



  csuaor46> vmstat 2 

  10 procs     
  memory            
  page            
  disk          
  faults      cpu r b w   swap  
  free  re  mf pi po fr de sr s6 s1 s1 s5   in   


  sy   cs us sy id 2 0 0  14536 14776  67 1514 15 
  228 957 62760 262 0 2 2 0 640 115 931 25 25 50 2 0 0 6179304 62416 36 
  1893 0 744 5500 56488 1126 0 4 4 0 988 6159 917 23 55 22 1 0 0 
  6177696 62600 20 908  0 1032 11808 56488 2488 0 6 6 0 917 3781 667 24 51 
  25 0 0 0 6181688 62960 89 1528 4 288 444 56488 37 0 6 6 0 1076 19029 
  862 23 54 23 1 0 0 6181336 64432 15 1269 0 140 576 56488 110 0 1 1 0 
  456 8550 493 14 46 40 0 0 0 6182376 63776 18 2976 4 368 1008 62760 
  328 0 8 8 0 594 6163 831 14 54 32 2 0 0 6180800 63072  9 1746 0 
  300 1296 62760 202 0 1 1 0 661 4441 693 12 65 23 0 0 0 6178120 62728 
  47 1311 4 612 2272 56488 464 0 3 2 0 829 5535 801 34 38 28 2 0 0 
  6179944 64616 36 1322 0 364 764 62760 70 0 0 0 0 996 4786  739 13 69 
  18 0 0 0 6183112 62560 40 856  4 340 1444 62760 339 0 1 2 0 822 
  4107 707 10 40 50csuaor46> swap -stotal: 2602216k bytes allocated + 
  19960k reserved = 2622176k used, 6177752k available
  I would assume swapping operations would 
  be included under "kernel"? The app also uses java, is there any way to 
  determine if Java is performing any wacky system calls?


   



  There are 3 databases on this box, which 
  has 4 Gb. of memory:


   



  csuaor46> ps -ef |grep ora_ |grep 


  smon  oracle   867     1  


  0   Aug 16 ?        0:13 


  ora_smon_tstrn  oracle   981     


  1  0   Aug 16 ?        0:14 


  ora_smon_tsdmo  oracle 19561     1  


  0   Aug 23 ?        0:35 


  ora_smon_tsprd


  The main db (tsprd) has an SGA of 1.7 
  Gb., the other 2 are ~180 Mb. each. - so that's ~2.1 Gb. There are only 
  49 connections to the 3 databases:


   



  csuaor46> ps -ef |grep LOCAL 


  |wc      49     


  447    3418


   



  We are not using MTS. Is there any way to 
  determine the amount of real memory that these dedicated connections are 
  using? I can't see how the whole 4 Gb. would be used, causing the system to 
  swap... ?


   



  The primary database:




    Sent: Tuesday, August 28, 2001 1:45 
    PM


    Subject: RE: OT : kernel using 75% of 
    CPU

    
    <SPAN 


    class=315352916-28082001>Hi Jerry,


    <SPAN 


    class=315352916-28082001> 


    <SPAN 


    class=315352916-28082001>Firstly, the kernel is not a process in the 
    conventional sense. It is basically a set of library functions. One of these 
    is the scheduler, which gets called every time slice, by the timer in the 
    hardware, in order to decide which actual process to run next. Responding to 
    interrupts is the only way in which a kernel could be considered to be 
    running. The kernel exists to provide services to processes, every time a 
    process makes a "system call", for example to perform I/O, this is invoking 
    a function within the kernel to actually "do" it - i.e. an application might 
    call read() then read() in the kernel would handle the business of talking 
    to the device driver and actually doing the reading of the data from the 
    disk (for example).


    <SPAN 


    class=315352916-28082001> 


    <SPAN 


    class=315352916-28082001>The CPU states line is showing you *where* the code 
    is running. If it's in "user" then the CPU is spending its time running code 
    in "user land" - probably computational code, stuff that's actually in the 
    application. If the state is "kernel", then it means that your application 
    is making lots of system calls, and the kernel level routines are doing the 
    work.


    <SPAN 


    class=315352916-28082001> 


    <SPAN 


    class=315352916-28082001>Incidentally, this is why Java is a good language 
    on the server - it does much of its real work in fast kernel space, and 
    little of it in the slow virtual machine. A busy Oracle will also spend a 
    bit of time in kernel space, doing I/O and networking, accessing shared 
    memory, etc. 


    <SPAN 


    class=315352916-28082001> 


    <SPAN 


    class=315352916-28082001>Looking at your "top" output, you have a high 
    system load, but your user processes aren't using much CPU, but kernel time 
    is high. This suggests that your processes are spending time waiting for the 
    kernel to do something or other for them, load being the size of the run 
    queue (all the processes that are ready to run but not actually on the CPU). 
    Are you actually experiencing performance problems? If so, you need to look 
    at what the system is doing using "sar" "vmstat" and "iostat". One thing to 
    watch out for is that "top" is a primitive tool. Notice how large all your 
    Oracle processes are? That is because top isn't smart enough to realize that 
    they're all connected to shared memory, it's counting each one as being 
    process size + SGA. So your processes and your memory in use don't add up. 
    Also top deals poorly with LWPs (threads) - are you using MTS? You could 
    simply be seeing threads stacking up as they wait for 
    network.


    <SPAN 


    class=315352916-28082001> 


    <SPAN 


    class=315352916-28082001>HTH,


    <SPAN 


    class=315352916-28082001> 


    <SPAN 


    class=315352916-28082001>g


    <SPAN 


    class=315352916-28082001> 
    

      <FONT face=Tahoma 
      size=2>-----Original Message-----From: Jerry C 
      [mailto:usidba_at_YAHOO.COM]Sent: Tuesday, August 28, 2001 4:20 
      PMTo: Multiple recipients of list ORACLE-LSubject: 
      OT : kernel using 75% of CPU
      Hi there,
       
      I have a Sun e4500, running Solaris 
      2.7 and Oracle 8.1.7.1.0. Everything looks normal from a database 
      perspective, but when I run "top" it show the kernel being very 
      hog-like:
       
      load averages: 14.38, 15.18, 
      15.18                                     
      07:16:21126 processes: 118 sleeping, 4 running, 4 on cpuCPU 
      states:  0.6% idle, 26.6% user, 72.8% kernel,  0.0% 
      iowait,  0.0% swapMemory: 4096M real, 63M free, 216M swap in use, 
      5310M swap free
       
        PID USERNAME THR PRI 
      NICE  SIZE   RES STATE   TIME    
      CPU COMMAND 2286 oracle     1   
      0    0 1844M 1814M run     9:44 13.90% 
      oracle11068 oracle     1   
      0    0 2056K 1536K cpu0    0:02  1.53% 
      top11333 oracle     1   
      0    0 1150M 1124M cpu1    0:01  1.39% 
      oracle 5944 oracle     1  
      40    0 1820M 1789M sleep  14:40  1.36% 
      oracle 4797 root       1  
      50    0 2112K 1248K sleep   6:01  1.36% 
      top11346 oracle     1   
      0    0  110M   92M cpu0    
      0:01  1.26% oracle11114 oracle     
      1   0    0 1009M  984M 
      cpu1    0:00  0.66% oracle11157 
      oracle     1   0    0 
      1009M  984M run     0:00  0.63% 
      oracle11368 oracle     1  
      33    0 1794M 1765M sleep   0:00  0.29% 
      oracle19558 oracle     1  
      60    0 1797M 1751M sleep  78:28  0.28% 
      oracle19554 oracle     1  
      60    0 1794M 1751M sleep  38:05  0.20% 
      oracle11366 oracle     1  
      55    0 1793M 1763M sleep   0:00  0.19% 
      oracle11292 oracle     1  
      26    2 2008K 1424K run     0:00  
      0.19% dsql
       
      Any ideas on what I, as a lowly DBA, 
      would be able to check? It's a bit out of my area and I'm 
      stumped...
       
       
      Thanks!
       
      <FONT face="Courier New" 




  size=2>Jerry
Received on Tue Aug 28 2001 - 16:04:22 CDT












Original text of this message













  HOME |
  ASK QUESTION |
  ADD INFO |
  SEARCH |
  E-MAIL US