Re: hugepages question

Call me lazy, but I just set soft and hard memlock limits to 'unlimited' and forget about it. :-)

On Fri, Feb 15, 2019 at 12:58 PM Jeff Chirco <backseatdba_at_gmail.com> wrote:

> Thanks Mark, that is interesting but haven't had luck yet.
>
> Ok one more question about memlock. In my /etc/security/limits.conf I
> have the following:
>
> oracle hard memlock 237262219
>
> oracle soft memlock 237262219
>
> However when I run ulimits -l I get 64 returned.
>
> ulimit -a returns:
>
> ulimit -a
>
> core file size (blocks, -c) 0
>
> data seg size (kbytes, -d) unlimited
>
> scheduling priority (-e) 0
>
> file size (blocks, -f) unlimited
>
> pending signals (-i) 1029648
>
> max locked memory (kbytes, -l) 64
>
> max memory size (kbytes, -m) unlimited
>
> open files (-n) 1024
>
> pipe size (512 bytes, -p) 8
>
> POSIX message queues (bytes, -q) 819200
>
> real-time priority (-r) 0
>
> stack size (kbytes, -s) 8192
>
> cpu time (seconds, -t) unlimited
>
> max user processes (-u) 1029648
>
> virtual memory (kbytes, -v) unlimited
>
> file locks (-x) unlimited
>
> Is memlock not getting set right? I am new to Linux so been learning a
> lot. Running Oracle Linux 7.
>
> Thanks,
> Jeff
>
> On Fri, Feb 15, 2019 at 8:21 AM Mark J. Bobak <mark_at_bobak.net> wrote:
>
>> Once you allocate hugepages, it prevents you using that memory for any
>> other purpose. If you're planning the use of the server, that's probably
>> fine
>>
>> Also, to add more to hugepages without a reboot, you can (theoretically)
>> do 'sudo sysctl -p' and then 'cat /proc/meminfo' to see if it fully
>> allocated all the newly requested hugepages. If not all hugepages were
>> allocated, you can do 'sudo sysctl -p' again, and check /proc/meminfo again.
>>
>> I remember, at least one time in the past, where I wanted to add a new
>> DB, and I couldn't afford a bounce of the server, due to other production
>> DBs on the server, where I wrote a simple script that did "grep
>> HugePages_Total /proc/meminfo |awk -F: '{ print $2 }'" and kept calling
>> 'sudo sysctl -p' over and over, until the output of the previous command
>> was equal to the value in '/etc/sysctl.conf'. It took a few minutes, but
>> it worked.
>>
>> -Mark
>>
>>
>> On Thu, Feb 14, 2019 at 11:53 AM Jose Rodriguez <jrodriguez2_at_pythian.com>
>> wrote:
>>
>>> I am curious to know why over allocating HP is a bad idea.
>>> I mean, if you know you will end up increasing the size of an SGA or the
>>> number of SGAs in the server, why not allocate HP ahead of time.
>>> Save some RAM for OS, some for PGA and the rest should go to SGA, that
>>> 'rest' should be in HP sooner or later, so do it sooner.
>>> Does it make sense? Or am I missing something?
>>>
>>> [image: Pythian] <http://www.pythian.com/>
>>> *Jose Rodriguez* | Oracle Project Engineer | [image: LinkedIn]
>>> <https://www.linkedin.com/company/pythian>
>>> *t* +1 613 565 8696 <+1+613+565+8696> *ext.* 1393
>>> *m* +34 607 55 49 91 <+34+607+55+49+91>
>>> jrodriguez2_at_pythian.com
>>> *www.pythian.com* <https://www.pythian.com/>
>>> [image: Pythian] <https://www.pythian.com/email-footer-click>
>>>
>>>
>>> On Thu, 14 Feb 2019 at 17:19, Jeff Chirco <backseatdba_at_gmail.com> wrote:
>>>
>>>> Hi Andy, yes I did reboot. Oh I originally set it to 81000 and then
>>>> rebooted. Yesterday I tried to increase it to 9100 without a reboot.
>>>>
>>>> Since it requires a reboot then that kind of is a bummer if I ever need
>>>> to add another database to a server. I'd have to bring everything down to
>>>> increase hugepages. I could over allocate hugepages but I read that is a
>>>> bad idea.
>>>>
>>>> Jeff
>>>>
>>>> On Thu, Feb 14, 2019 at 8:05 AM Andy Wattenhofer <watt0012_at_umn.edu>
>>>> wrote:
>>>>
>>>>> Did you reboot after the last change to hugepages?
>>>>>
>>>>> [root]# grep nr_hugepages /etc/sysctl.conf
>>>>>> vm.nr_hugepages=*91000*
>>>>>>
>>>>> [...]
>>>>>
>>>>>> [root]# cat /proc/meminfo
>>>>>> HugePages_Total: *81793*
>>>>>>
>>>>> HugePages_Free: 8
>>>>>> HugePages_Rsvd: 7
>>>>>> HugePages_Surp: 0
>>>>>>
>>>>>
>>>>> Yes, you must reboot for 91000 to take effect. The huge page RAM is
>>>>> reserved at boot time.
>>>>>
>>>>> Andy
>>>>>
>>>>>
>>>>> On Thu, Feb 14, 2019 at 9:15 AM Jeff Chirco <backseatdba_at_gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks for the replies. Forgot to mention this is Oracle Linux 7
>>>>>> running 12.2.0.1. These are output from my Development server but
>>>>>> production is identicial except hugepage size is lower cause it has less
>>>>>> database.
>>>>>> Development and Production both have 256gb of RAM.
>>>>>> I also noticed when I first set huegpages using the below command it
>>>>>> increased hugepages but not to the amount I tried to set it to. Not until a
>>>>>> server reboot did it get set the full amount. If I need to increase it
>>>>>> again I would prefer to not have to reboot the server.
>>>>>> # sysctl -w vm.nr_hugepages=value
>>>>>>
>>>>>> Regarding hugepages_settings.sh I did follow Tim's notes but I ended
>>>>>> up finding an updated script under Doc ID 401749.1
>>>>>>
>>>>>> [root]# grep memlock /etc/security/limits.conf
>>>>>> # - memlock - max locked-in-memory address space (KB)
>>>>>> oracle hard memlock 237265300
>>>>>> # oracle-rdbms-server-11gR2-preinstall setting for memlock hard limit
>>>>>> is maximum of 128GB on x86_64 or 3GB on x86 OR 90 % of RAM
>>>>>> oracle soft memlock 237265300
>>>>>>
>>>>>> [root]# grep nr_hugepages /etc/sysctl.conf
>>>>>> vm.nr_hugepages=91000
>>>>>>
>>>>>>
>>>>>> [root]# ulimit -l
>>>>>> 64
>>>>>>
>>>>>> [root]# ulimit -a
>>>>>> core file size (blocks, -c) 0
>>>>>> data seg size (kbytes, -d) unlimited
>>>>>> scheduling priority (-e) 0
>>>>>> file size (blocks, -f) unlimited
>>>>>> pending signals (-i) 1029648
>>>>>> max locked memory (kbytes, -l) 64
>>>>>> max memory size (kbytes, -m) unlimited
>>>>>> open files (-n) 1024
>>>>>> pipe size (512 bytes, -p) 8
>>>>>> POSIX message queues (bytes, -q) 819200
>>>>>> real-time priority (-r) 0
>>>>>> stack size (kbytes, -s) 8192
>>>>>> cpu time (seconds, -t) unlimited
>>>>>> max user processes (-u) 1029648
>>>>>> virtual memory (kbytes, -v) unlimited
>>>>>> file locks (-x) unlimited
>>>>>>
>>>>>>
>>>>>> Alert log from one of my databases
>>>>>> **********************************************************************
>>>>>> 2019-02-12T21:26:58.786205-08:00
>>>>>> Dump of system resources acquired for SHARED GLOBAL AREA (SGA)
>>>>>>
>>>>>> 2019-02-12T21:26:58.790788-08:00
>>>>>> Per process system memlock (soft) limit = 226G
>>>>>> 2019-02-12T21:26:58.793885-08:00
>>>>>> Expected per process system memlock (soft) limit to lock
>>>>>> SHARED GLOBAL AREA (SGA) into memory: 8196M
>>>>>> 2019-02-12T21:26:58.800179-08:00
>>>>>> Available system pagesizes:
>>>>>> 4K, 2048K
>>>>>> 2019-02-12T21:26:58.806848-08:00
>>>>>> Supported system pagesize(s):
>>>>>> 2019-02-12T21:26:58.809846-08:00
>>>>>> PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
>>>>>> 2019-02-12T21:26:58.813115-08:00
>>>>>> 2048K 65468 4098 4098
>>>>>> NONE
>>>>>> 2019-02-12T21:26:58.816156-08:00
>>>>>> Reason for not supporting certain system pagesizes:
>>>>>> 2019-02-12T21:26:58.819060-08:00
>>>>>> 4K - Large pagesizes only
>>>>>> 2019-02-12T21:26:58.822058-08:00
>>>>>> **********************************************************************
>>>>>>
>>>>>>
>>>>>>
>>>>>> [root]# ipcs -m
>>>>>>
>>>>>> ------ Shared Memory Segments --------
>>>>>> key shmid owner perms bytes nattch
>>>>>> status
>>>>>> 0x00000000 589824 oracle 600 10485760 78
>>>>>> 0x00000000 622593 oracle 600 3372220416 78
>>>>>> 0x00000000 655362 oracle 600 8388608 78
>>>>>> 0x76c55318 688131 oracle 600 2097152 78
>>>>>> 0x00000000 11862020 oracle 600 10485760 120
>>>>>> 0x00000000 11894789 oracle 600 3204448256 120
>>>>>> 0x00000000 11927558 oracle 600 8388608 120
>>>>>> 0x34ef509c 11960327 oracle 600 2097152 120
>>>>>> 0x00000000 11993096 oracle 600 10485760 95
>>>>>> 0x00000000 12025865 oracle 600 2298478592 95
>>>>>> 0x00000000 12058634 oracle 600 8388608 95
>>>>>> 0x507a78a4 12091403 oracle 600 2097152 95
>>>>>> 0x00000000 12124172 oracle 600 10485760 207
>>>>>> 0x00000000 12156941 oracle 600 6425673728 207
>>>>>> 0x00000000 12189710 oracle 600 8388608 207
>>>>>> 0x866b8aa0 12222479 oracle 600 2097152 207
>>>>>> 0x00000000 12255248 oracle 600 12582912 81
>>>>>> 0x00000000 12288017 oracle 600 8573157376 81
>>>>>> 0x00000000 12320786 oracle 600 6291456 81
>>>>>> 0x0c51f55c 12353555 oracle 600 2097152 81
>>>>>> 0x00000000 12386324 oracle 600 12582912 79
>>>>>> 0x00000000 12419093 oracle 600 8573157376 79
>>>>>> 0x00000000 12451862 oracle 600 6291456 79
>>>>>> 0x7afafd64 12484631 oracle 600 2097152 79
>>>>>> 0x00000000 12779544 oracle 600 12582912 90
>>>>>> 0x00000000 12812313 oracle 600 8573157376 90
>>>>>> 0x00000000 12845082 oracle 600 6291456 90
>>>>>> 0xdde48098 12877851 oracle 600 2097152 90
>>>>>> 0x00000000 12910620 oracle 600 12582912 86
>>>>>> 0x00000000 12943389 oracle 600 8573157376 86
>>>>>> 0x00000000 12976158 oracle 600 6291456 86
>>>>>> 0xbae4e74c 13008927 oracle 600 2097152 86
>>>>>> 0x00000000 13303840 oracle 600 12582912 85
>>>>>> 0x00000000 13336609 oracle 600 8573157376 85
>>>>>> 0x00000000 13369378 oracle 600 6291456 85
>>>>>> 0x4c7ac944 13402147 oracle 600 2097152 85
>>>>>> 0x00000000 13434916 oracle 600 12582912 86
>>>>>> 0x00000000 13467685 oracle 600 8573157376 86
>>>>>> 0x00000000 13500454 oracle 600 6291456 86
>>>>>> 0xf06d661c 13533223 oracle 600 2097152 86
>>>>>> 0x00000000 13828136 oracle 600 12582912 87
>>>>>> 0x00000000 13860905 oracle 600 8573157376 87
>>>>>> 0x00000000 13893674 oracle 600 6291456 87
>>>>>> 0xdb5acf68 13926443 oracle 600 2097152 87
>>>>>> 0x00000000 13959212 oracle 600 12582912 87
>>>>>> 0x00000000 13991981 oracle 600 8573157376 87
>>>>>> 0x00000000 14024750 oracle 600 6291456 87
>>>>>> 0xcdd9f634 14057519 oracle 600 2097152 87
>>>>>> 0x00000000 14352432 oracle 600 10485760 102
>>>>>> 0x00000000 14385201 oracle 600 6425673728 102
>>>>>> 0x00000000 14417970 oracle 600 8388608 102
>>>>>> 0x4bd6517c 14450739 oracle 600 2097152 102
>>>>>> 0x00000000 14483508 oracle 600 10485760 116
>>>>>> 0x00000000 14516277 oracle 600 6425673728 116
>>>>>> 0x00000000 14549046 oracle 600 8388608 116
>>>>>> 0x33ec8074 14581815 oracle 600 2097152 116
>>>>>> 0x00000000 14876728 oracle 600 12582912 87
>>>>>> 0x00000000 14909497 oracle 600 8573157376 87
>>>>>> 0x00000000 14942266 oracle 600 6291456 87
>>>>>> 0xb40541dc 14975035 oracle 600 2097152 87
>>>>>> 0x00000000 1179975740 oracle 600 12582912 90
>>>>>> 0x00000000 1180008509 oracle 600 8455716864 90
>>>>>> 0x00000000 1180041278 oracle 600 34359738368 90
>>>>>> 0x00000000 1180074047 oracle 600 123731968 90
>>>>>> 0x00000000 15401024 oracle 600 12582912 91
>>>>>> 0x00000000 15433793 oracle 600 8573157376 91
>>>>>> 0x00000000 15466562 oracle 600 6291456 91
>>>>>> 0x36329238 15499331 oracle 600 2097152 91
>>>>>> 0x00000000 15532100 oracle 600 12582912 118
>>>>>> 0x00000000 15564869 oracle 600 8573157376 118
>>>>>> 0x00000000 15597638 oracle 600 6291456 118
>>>>>> 0xa0206b88 15630407 oracle 600 2097152 118
>>>>>> 0x00000000 16449608 oracle 600 12582912 86
>>>>>> 0x00000000 16482377 oracle 600 8573157376 86
>>>>>> 0x00000000 16515146 oracle 600 6291456 86
>>>>>> 0xc402bb18 16547915 oracle 600 2097152 86
>>>>>> 0x00000000 16318540 oracle 600 12582912 87
>>>>>> 0x00000000 16351309 oracle 600 8573157376 87
>>>>>> 0x00000000 16384078 oracle 600 6291456 87
>>>>>> 0x261e431c 16416847 oracle 600 2097152 87
>>>>>> 0x00000000 16187472 oracle 600 12582912 168
>>>>>> 0x00000000 16220241 oracle 600 20937965568 168
>>>>>> 0x00000000 16253010 oracle 600 56623104 168
>>>>>> 0x35b8fa94 16285779 oracle 600 2097152 168
>>>>>> 0x00000000 547127380 oracle 600 12582912 85
>>>>>> 0x00000000 547160149 oracle 600 251658240 85
>>>>>> 0x00000000 547192918 oracle 600 8321499136 85
>>>>>> 0x00000000 547225687 oracle 600 4595712 85
>>>>>> 0x9cd83398 547258456 oracle 600 40960 85
>>>>>> 0x00000000 792199257 oracle 600 10485760 106
>>>>>> 0x00000000 792232026 oracle 600 1644167168 106
>>>>>> 0x00000000 792264795 oracle 600 4781506560 106
>>>>>> 0x00000000 792297564 oracle 600 8388608 106
>>>>>> 0xc78a1c8c 792330333 oracle 600 32768 106
>>>>>> 0x6762e034 1180106846 oracle 600 28672 90
>>>>>>
>>>>>> [root]# cat /proc/meminfo
>>>>>> MemTotal: 263623524 kB
>>>>>> MemFree: 2169856 kB
>>>>>> MemAvailable: 25248596 kB
>>>>>> Buffers: 28 kB
>>>>>> Cached: 68234496 kB
>>>>>> SwapCached: 4392 kB
>>>>>> Active: 56288772 kB
>>>>>> Inactive: 32305392 kB
>>>>>> Active(anon): 38139416 kB
>>>>>> Inactive(anon): 28588332 kB
>>>>>> Active(file): 18149356 kB
>>>>>> Inactive(file): 3717060 kB
>>>>>> Unevictable: 0 kB
>>>>>> Mlocked: 0 kB
>>>>>> SwapTotal: 16777212 kB
>>>>>> SwapFree: 16732780 kB
>>>>>> Dirty: 1020 kB
>>>>>> Writeback: 0 kB
>>>>>> AnonPages: 20491988 kB
>>>>>> Mapped: 10605612 kB
>>>>>> Shmem: 46232620 kB
>>>>>> Slab: 2937544 kB
>>>>>> SReclaimable: 2197872 kB
>>>>>> SUnreclaim: 739672 kB
>>>>>> KernelStack: 68544 kB
>>>>>> PageTables: 1451148 kB
>>>>>> NFS_Unstable: 0 kB
>>>>>> Bounce: 0 kB
>>>>>> WritebackTmp: 0 kB
>>>>>> CommitLimit: 64832940 kB
>>>>>> Committed_AS: 93046052 kB
>>>>>> VmallocTotal: 34359738367 kB
>>>>>> VmallocUsed: 772444 kB
>>>>>> VmallocChunk: 34358900732 kB
>>>>>> HardwareCorrupted: 0 kB
>>>>>> AnonHugePages: 0 kB
>>>>>> CmaTotal: 0 kB
>>>>>> CmaFree: 0 kB
>>>>>> HugePages_Total: 81793
>>>>>> HugePages_Free: 8
>>>>>> HugePages_Rsvd: 7
>>>>>> HugePages_Surp: 0
>>>>>> Hugepagesize: 2048 kB
>>>>>> DirectMap4k: 514768 kB
>>>>>> DirectMap2M: 41201664 kB
>>>>>> DirectMap1G: 228589568 kB
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 14, 2019 at 6:50 AM Jeff Chirco <backseatdba_at_gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Ooops sorry yes Oracle Linux 7.4
>>>>>>>
>>>>>>> On Wed, Feb 13, 2019 at 4:31 PM Paul Drake <bdbafh_at_gmail.com> wrote:
>>>>>>>
>>>>>>>> OS info might be relevant.
>>>>>>>>
>>>>>>>> Assuming Linux:
>>>>>>>>
>>>>>>>> ipcs -m
>>>>>>>> cat /proc/meminfo
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Feb 13, 2019, 7:12 PM Jeff Chirco <backseatdba_at_gmail.com
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Ok I just recently enabled hugepages on Production and
>>>>>>>>> Development. Everything is going good but I noticed that on my development
>>>>>>>>> environment which consists of a bunch of thin clones, when I remove and
>>>>>>>>> recreate the same clone the database fails to create, I am getting:
>>>>>>>>>
>>>>>>>>> ORA-27137: unable to allocate large pages to create a shared memory segment
>>>>>>>>>
>>>>>>>>> I have the parameter *use_large_pages=ONLY*
>>>>>>>>> So it makes sense I would get this error when there is not enough
>>>>>>>>> pages. However I just removed this same database and am recreating it. I
>>>>>>>>> would think that would free it up but apparently not. Does pages not get
>>>>>>>>> released immediately when a database is removed? Is there something I
>>>>>>>>> should run to free it? Or just not have use_large_pages=ONLY and leave it
>>>>>>>>> at TRUE on Dev? I had to set it to TRUE to get this database up.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Jeff
>>>>>>>>>
>>>>>>>>
>>> --
>>>
>>>
>>>
>>>

--
http://www.freelists.org/webpage/oracle-l