무작위로(약 한 달에 한 번) OOM 문제가 발생하지만 여유 메모리가 충분한 것 같습니다.
원인도 해결방법도 모르겠습니다...
vm.min_free_kbytes = 90112
Linux xxx-alpha 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux
slabtop
Active / Total Objects (% used) : 45902519 / 54189870 (84.7%)
Active / Total Slabs (% used) : 1260240 / 1260240 (100.0%)
Active / Total Caches (% used) : 151 / 187 (80.7%)
Active / Total Size (% used) : 9241442.17K / 10856556.48K (85.1%)
Minimum / Average / Maximum Object : 0.01K / 0.20K / 23.25K
OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
43615236 37957660 87% 0.19K 1038467 42 8307736K dentry
2180640 1579066 72% 0.07K 38940 56 155760K range_seg_cache
706048 538395 76% 0.01K 1379 512 5516K kmalloc-8
644992 395397 61% 0.03K 5039 128 20156K kmalloc-32
635150 294932 46% 0.32K 12963 49 207408K arc_buf_hdr_t_full
546840 332510 60% 0.09K 13020 42 52080K kmalloc-96
dmesg
[Sun Jan 10 00:05:35 2021] elasticsearch[7 invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
[Sun Jan 10 00:05:35 2021] elasticsearch[7 cpuset=67e9bddb581923bf7602fbcce820867ff13dd19355d458112bee6300e0a2cbd8 mems_allowed=0-1
[Sun Jan 10 00:05:35 2021] CPU: 45 PID: 9599 Comm: elasticsearch[7 Tainted: P OE 4.19.0-6-amd64 #1 Debian 4.19.67-2+deb10u2
[Sun Jan 10 00:05:35 2021] Hardware name: Ericsson CRU 0101/S2BS-MB, BIOS S2B_3B20 05/30/2019
[Sun Jan 10 00:05:35 2021] Call Trace:
[Sun Jan 10 00:05:35 2021] dump_stack+0x5c/0x80
[Sun Jan 10 00:05:35 2021] dump_header+0x6b/0x283
[Sun Jan 10 00:05:35 2021] ? do_try_to_free_pages+0x2ec/0x370
[Sun Jan 10 00:05:35 2021] oom_kill_process.cold.30+0xb/0x1cf
[Sun Jan 10 00:05:35 2021] ? oom_badness+0x23/0x140
[Sun Jan 10 00:05:35 2021] out_of_memory+0x1a5/0x430
[Sun Jan 10 00:05:35 2021] __alloc_pages_slowpath+0xbd8/0xcb0
[Sun Jan 10 00:05:35 2021] __alloc_pages_nodemask+0x28b/0x2b0
[Sun Jan 10 00:05:35 2021] filemap_fault+0x3bd/0x780
[Sun Jan 10 00:05:35 2021] ? filemap_map_pages+0x1ed/0x3a0
[Sun Jan 10 00:05:35 2021] __do_fault+0x36/0x130
[Sun Jan 10 00:05:35 2021] __handle_mm_fault+0xe6c/0x1270
[Sun Jan 10 00:05:35 2021] handle_mm_fault+0xd6/0x200
[Sun Jan 10 00:05:35 2021] __do_page_fault+0x249/0x4f0
[Sun Jan 10 00:05:35 2021] ? page_fault+0x8/0x30
[Sun Jan 10 00:05:35 2021] page_fault+0x1e/0x30
[Sun Jan 10 00:05:35 2021] RIP: 0033:0x7f68612bc435
[Sun Jan 10 00:05:35 2021] Code: 1f 00 00 49 c1 e0 03 41 8b 48 18 45 8b 48 1c 41 3b c9 0f 8d 47 21 00 00 4d 8b 50 10 8b d9 ff c3 41 89 58 18 49 8b fa 4c 63 d1 <46> 0f be 14 17 45 85 d2 0f 8c 07 0d 00 00 41 8b ca 83 e1 01 41 88
[Sun Jan 10 00:05:35 2021] RSP: 002b:00007e876f44e050 EFLAGS: 00010207
[Sun Jan 10 00:05:35 2021] RAX: 0000000000000002 RBX: 00000000000e37a6 RCX: 00000000000e37a5
[Sun Jan 10 00:05:35 2021] RDX: 0000000684825b90 RSI: 000000000032bce7 RDI: 00007f625cad5441
[Sun Jan 10 00:05:35 2021] RBP: 00000000d3887259 R08: 0000000684825bd0 R09: 00000000002f8d73
[Sun Jan 10 00:05:35 2021] R10: 00000000000e37a5 R11: 0000000684825b90 R12: 0000000000000000
[Sun Jan 10 00:05:35 2021] R13: 00000006829fb780 R14: 00000000d0904b72 R15: 00007f66dc05f800
[Sun Jan 10 00:05:35 2021] Mem-Info:
[Sun Jan 10 00:05:35 2021] active_anon:12218942 inactive_anon:93895 isolated_anon:0
active_file:1203 inactive_file:1345 isolated_file:6
unevictable:0 dirty:15 writeback:21 unstable:0
slab_reclaimable:1710574 slab_unreclaimable:1562977
mapped:26078 shmem:177912 pagetables:59851 bounce:0
free:49188 free_pcp:9835 free_cma:0
[Sun Jan 10 00:05:35 2021] Node 0 active_anon:31064964kB inactive_anon:48776kB active_file:284kB inactive_file:392kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:18116kB dirty:4kB writeback:4kB shmem:128568kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Sun Jan 10 00:05:35 2021] Node 1 active_anon:17810804kB inactive_anon:326804kB active_file:4528kB inactive_file:4988kB unevictable:0kB isolated(anon):0kB isolated(file):24kB mapped:86196kB dirty:56kB writeback:80kB shmem:583080kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[Sun Jan 10 00:05:35 2021] Node 0 DMA free:15904kB min:20kB low:32kB high:44kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15904kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 1848 32052 32052 32052
[Sun Jan 10 00:05:35 2021] Node 0 DMA32 free:123288kB min:2556kB low:4420kB high:6284kB active_anon:1651524kB inactive_anon:2372kB active_file:116kB inactive_file:56kB unevictable:0kB writepending:0kB present:1959556kB managed:1892668kB mlocked:0kB kernel_stack:212kB pagetables:2500kB bounce:0kB free_pcp:248kB local_pcp:0kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 0 30203 30203 30203
[Sun Jan 10 00:05:35 2021] Node 0 Normal free:22820kB min:42348kB low:73276kB high:104204kB active_anon:29413440kB inactive_anon:46404kB active_file:156kB inactive_file:336kB unevictable:0kB writepending:8kB present:31457280kB managed:30928584kB mlocked:0kB kernel_stack:21416kB pagetables:78456kB bounce:0kB free_pcp:6260kB local_pcp:0kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 0 0 0 0
[Sun Jan 10 00:05:35 2021] Node 1 Normal free:34740kB min:45184kB low:78184kB high:111184kB active_anon:17810804kB inactive_anon:326804kB active_file:5176kB inactive_file:5044kB unevictable:0kB writepending:136kB present:33554432kB managed:33004524kB mlocked:0kB kernel_stack:47908kB pagetables:158448kB bounce:0kB free_pcp:32832kB local_pcp:1324kB free_cma:0kB
[Sun Jan 10 00:05:35 2021] lowmem_reserve[]: 0 0 0 0 0
[Sun Jan 10 00:05:35 2021] Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 3*64kB (U) 0*128kB 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
[Sun Jan 10 00:05:35 2021] Node 0 DMA32: 99*4kB (UMH) 15*8kB (UMH) 3483*16kB (UMH) 2052*32kB (UMEH) 3*64kB (MH) 1*128kB (H) 3*256kB (H) 1*512kB (H) 0*1024kB 0*2048kB 0*4096kB = 123508kB
[Sun Jan 10 00:05:35 2021] Node 0 Normal: 882*4kB (UMEH) 33*8kB (MEH) 34*16kB (ME) 751*32kB (UME) 3*64kB (UH) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 28560kB
[Sun Jan 10 00:05:35 2021] Node 1 Normal: 8333*4kB (MH) 26*8kB (H) 1*16kB (H) 1*32kB (H) 3*64kB (H) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 33780kB
[Sun Jan 10 00:05:35 2021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Sun Jan 10 00:05:35 2021] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sun Jan 10 00:05:35 2021] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Sun Jan 10 00:05:35 2021] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Sun Jan 10 00:05:35 2021] 181392 total pagecache pages
[Sun Jan 10 00:05:35 2021] 0 pages in swap cache
[Sun Jan 10 00:05:35 2021] Swap cache stats: add 0, delete 0, find 0/0
[Sun Jan 10 00:05:35 2021] Free swap = 0kB
[Sun Jan 10 00:05:35 2021] Total swap = 0kB
[Sun Jan 10 00:05:35 2021] 16746816 pages RAM
[Sun Jan 10 00:05:35 2021] 0 pages HighMem/MovableOnly
[Sun Jan 10 00:05:35 2021] 286396 pages reserved
[Sun Jan 10 00:05:35 2021] 0 pages hwpoisoned
[Sun Jan 10 00:05:35 2021] Tasks state (memory values in pages):
[Sun Jan 10 00:05:35 2021] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[Sun Jan 10 00:05:35 2021] [ 611] 0 611 60450 40093 503808 0 0 systemd-journal
[Sun Jan 10 00:05:35 2021] [ 635] 0 635 5616 348 61440 0 -1000 systemd-udevd
[Sun Jan 10 00:05:35 2021] [ 814] 0 814 770 56 45056 0 0 mdadm
[Sun Jan 10 00:05:35 2021] [ 9990] 0 9990 41684 221 69632 0 0 zed
[Sun Jan 10 00:05:35 2021] [ 9991] 0 9991 2557 73 61440 0 0 cron
[Sun Jan 10 00:05:35 2021] [ 9993] 104 9993 2222 187 57344 0 -900 dbus-daemon
[Sun Jan 10 00:05:35 2021] [ 9994] 0 9994 4991 386 77824 0 0 systemd-logind
[Sun Jan 10 00:05:35 2021] [ 10013] 0 10013 200898 2220 65536 0 0 supervisord
[Sun Jan 10 00:05:35 2021] [ 10020] 0 10020 28023 1915 114688 0 0 unattended-upgr
[Sun Jan 10 00:05:35 2021] [ 10028] 0 10028 1405091 6815 843776 0 0 containerd
[Sun Jan 10 00:05:35 2021] [ 10033] 0 10033 1307161 15321 995328 0 -500 dockerd
[Sun Jan 10 00:05:35 2021] [ 10040] 0 10040 3963 236 69632 0 -1000 sshd
[Sun Jan 10 00:05:35 2021] [ 10074] 0 10074 4934 1407 73728 0 0 haproxy
[Sun Jan 10 00:05:35 2021] [ 12823] 0 12823 1403 28 45056 0 0 agetty
[Sun Jan 10 00:05:35 2021] [ 13945] 107 13945 4935 210 81920 0 0 exim4
[Sun Jan 10 00:05:35 2021] [ 19622] 0 19622 5382 445 81920 0 0 systemd
[Sun Jan 10 00:05:35 2021] [ 19623] 0 19623 5779 630 73728 0 0 (sd-pam)
[Sun Jan 10 00:05:35 2021] [ 24193] 0 24193 31559 4572 143360 0 0 prom-node-expor
[Sun Jan 10 00:05:35 2021] [ 9763] 0 9763 61147 1433 65536 0 0 nginx
[Sun Jan 10 00:05:35 2021] [ 4864] 0 4864 58959 222 94208 0 0 polkitd
[Sun Jan 10 00:05:35 2021] [ 19619] 0 19619 155613 481 126976 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 19633] 0 19633 27276 212 73728 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 19686] 472 19686 46091 7685 315392 0 0 grafana-server
[Sun Jan 10 00:05:35 2021] [ 34921] 0 34921 820113 471 462848 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 34929] 0 34929 26924 202 73728 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 34948] 65534 34948 5391852 43302 21938176 0 0 prometheus
[Sun Jan 10 00:05:35 2021] [ 12393] 108 12393 19145 168 65536 0 0 ntpd
[Sun Jan 10 00:05:35 2021] [ 5862] 0 5862 4947 2284 90112 0 0 tmux: server
[Sun Jan 10 00:05:35 2021] [ 5863] 0 5863 2209 151 61440 0 0 bash
[Sun Jan 10 00:05:35 2021] [ 33048] 0 33048 2249 215 57344 0 0 bash
[Sun Jan 10 00:05:35 2021] [ 29476] 0 29476 2201 154 53248 0 0 bash
[Sun Jan 10 00:05:35 2021] [ 31493] 0 31493 597 46 45056 0 0 mysqld_safe
[Sun Jan 10 00:05:35 2021] [ 45444] 0 45444 5315 1563 86016 0 0 mysql
[Sun Jan 10 00:05:35 2021] [ 13606] 0 13606 2209 156 61440 0 0 bash
[Sun Jan 10 00:05:35 2021] [ 35429] 0 35429 2179 152 49152 0 0 bash
[Sun Jan 10 00:05:35 2021] [ 26987] 0 26987 137180 482 118784 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 27040] 0 27040 985866 1166 544768 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 27049] 0 27049 27276 940 86016 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 27074] 0 27074 4947 74 81920 0 0 wrapper
[Sun Jan 10 00:05:35 2021] [ 28817] 0 28817 1098 22 53248 0 0 runsvdir
[Sun Jan 10 00:05:35 2021] [ 29114] 0 29114 1060 19 49152 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 29117] 0 29117 1096 20 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 29120] 0 29120 16378 178 167936 0 0 sshd
[Sun Jan 10 00:05:35 2021] [ 34657] 0 34657 1060 18 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 34660] 997 34660 18749 2493 143360 0 0 redis-server
[Sun Jan 10 00:05:35 2021] [ 35797] 0 35797 1096 22 57344 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 37303] 0 37303 1060 18 49152 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 37305] 998 37305 760003 445 475136 0 0 gitaly-wrapper
[Sun Jan 10 00:05:35 2021] [ 37311] 998 37311 759550 23379 1101824 0 0 gitaly
[Sun Jan 10 00:05:35 2021] [ 37587] 998 37587 728803 17562 888832 0 0 ruby
[Sun Jan 10 00:05:35 2021] [ 37588] 998 37588 778727 39586 1073152 0 0 ruby
[Sun Jan 10 00:05:35 2021] [ 37595] 0 37595 1096 21 57344 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 39290] 0 39290 1060 19 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 39299] 996 39299 12338 2782 131072 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39329] 996 39329 12392 920 110592 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39330] 996 39330 12374 831 110592 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39331] 996 39331 12338 543 102400 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39332] 996 39332 12473 1007 118784 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39333] 996 39333 8152 553 94208 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39334] 996 39334 12451 709 118784 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 40268] 0 40268 1096 20 49152 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 41373] 0 41373 1060 18 49152 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 41375] 998 41375 330686 207154 2560000 0 0 bundle
[Sun Jan 10 00:05:35 2021] [ 42757] 0 42757 1096 22 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 43230] 0 43230 1060 19 45056 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 43232] 998 43232 34288 3288 274432 0 0 ruby
[Sun Jan 10 00:05:35 2021] [ 43261] 998 43261 891610 247976 6795264 0 0 bundle
[Sun Jan 10 00:05:35 2021] [ 44871] 0 44871 1096 21 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 45716] 0 45716 1060 18 49152 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 46557] 0 46557 1096 20 57344 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 47976] 0 47976 1060 18 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 47978] 0 47978 11158 287 114688 0 0 nginx
[Sun Jan 10 00:05:35 2021] [ 48665] 0 48665 1096 20 57344 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 1841] 0 1841 1060 18 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 2211] 0 2211 1096 20 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 4501] 0 4501 1060 18 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 4503] 998 4503 237878 114014 1216512 0 0 gitlab-exporter
[Sun Jan 10 00:05:35 2021] [ 5326] 0 5326 1096 21 57344 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 6712] 0 6712 1060 19 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 7193] 0 7193 1096 21 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 9114] 0 9114 1060 18 49152 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 11360] 0 11360 1096 23 57344 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 13794] 0 13794 1060 19 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 15522] 0 15522 1096 21 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 17228] 0 17228 1060 19 53248 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 18181] 0 18181 1096 22 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 20227] 0 20227 1060 18 49152 0 0 runsv
[Sun Jan 10 00:05:35 2021] [ 21159] 0 21159 1096 21 53248 0 0 svlogd
[Sun Jan 10 00:05:35 2021] [ 23043] 996 23043 15811 3066 151552 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 23048] 996 23048 15008 1972 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 23052] 996 23052 15801 3037 151552 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 29245] 996 29245 14829 1534 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 30217] 996 30217 14830 1535 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 30363] 996 30363 15482 2791 147456 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 31925] 998 31925 727762 7223 606208 0 0 gitlab-workhors
[Sun Jan 10 00:05:35 2021] [ 32066] 997 32066 640030 2595 475136 0 0 redis_exporter
[Sun Jan 10 00:05:35 2021] [ 32276] 992 32276 1962869 75117 2211840 0 0 prometheus
[Sun Jan 10 00:05:35 2021] [ 32438] 992 32438 696610 2725 503808 0 0 alertmanager
[Sun Jan 10 00:05:35 2021] [ 32620] 996 32620 691549 4155 540672 0 0 postgres_export
[Sun Jan 10 00:05:35 2021] [ 32626] 996 32626 21203 6899 180224 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 32693] 992 32693 1181528 5796 720896 0 0 grafana-server
[Sun Jan 10 00:05:35 2021] [ 33814] 0 33814 4939 59 86016 0 0 gitlab-ctl
[Sun Jan 10 00:05:35 2021] [ 33828] 0 33828 37816 6500 335872 0 0 omnibus-ctl
[Sun Jan 10 00:05:35 2021] [ 34377] 0 34377 1126 18 57344 0 0 sh
[Sun Jan 10 00:05:35 2021] [ 34380] 0 34380 1595 20 53248 0 0 xargs
[Sun Jan 10 00:05:35 2021] [ 34381] 0 34381 1528 31 57344 0 0 tail
[Sun Jan 10 00:05:35 2021] [ 35947] 0 35947 2179 145 49152 0 0 bash
[Sun Jan 10 00:05:35 2021] [ 23562] 998 23562 9032150 7162562 62947328 0 0 mysqld
[Sun Jan 10 00:05:35 2021] [ 44618] 0 44618 137180 481 122880 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 44627] 0 44627 27276 195 77824 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 44706] 2000 44706 9185087 552797 6643712 0 0 java
[Sun Jan 10 00:05:35 2021] [ 41452] 0 41452 155613 480 126976 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 41460] 0 41460 27276 338 73728 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 41475] 0 41475 27276 255 73728 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 41494] 999 41494 327076 55842 933888 0 0 mongod
[Sun Jan 10 00:05:35 2021] [ 41503] 1000 41503 422008290 1906347 96919552 0 0 java
[Sun Jan 10 00:05:35 2021] [ 43137] 0 43137 137180 481 122880 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 43150] 0 43150 230017 1307 176128 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 43164] 0 43164 155613 479 131072 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 43178] 0 43178 137180 483 131072 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 43192] 0 43192 192959 1068 159744 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 43208] 0 43208 155613 478 135168 0 -500 docker-proxy
[Sun Jan 10 00:05:35 2021] [ 43234] 0 43234 27276 998 77824 0 -999 containerd-shim
[Sun Jan 10 00:05:35 2021] [ 43357] 1100 43357 571 20 45056 0 0 tini
[Sun Jan 10 00:05:35 2021] [ 43798] 1100 43798 1372 78 49152 0 0 docker-entrypoi
[Sun Jan 10 00:05:35 2021] [ 43828] 1100 43828 11417937 1054338 12394496 0 0 java
[Sun Jan 10 00:05:35 2021] [ 23584] 106 23584 5178 1676 77824 0 0 haproxy
[Sun Jan 10 00:05:35 2021] [ 23691] 33 23691 61703 1991 73728 0 0 nginx
[Sun Jan 10 00:05:35 2021] [ 23692] 33 23692 61653 1844 73728 0 0 nginx
[Sun Jan 10 00:05:35 2021] [ 24208] 0 24208 180714 7702 192512 0 0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [ 24216] 0 24216 180650 6978 204800 0 0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [ 24243] 0 24243 180842 7627 200704 0 0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [ 24256] 0 24256 180906 7859 204800 0 0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [ 24279] 0 24279 180778 7917 204800 0 0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [ 24303] 0 24303 181146 10773 221184 0 0 prom-pushgatewa
[Sun Jan 10 00:05:35 2021] [ 48493] 998 48493 460417 215657 3211264 0 0 bundle
[Sun Jan 10 00:05:35 2021] [ 48615] 996 48615 14829 1524 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 20931] 998 20931 464001 225770 3223552 0 0 bundle
[Sun Jan 10 00:05:35 2021] [ 20977] 996 20977 14829 1493 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 46187] 998 46187 447326 220269 3133440 0 0 bundle
[Sun Jan 10 00:05:35 2021] [ 46206] 996 46206 14829 1550 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 27018] 998 27018 452737 226133 3174400 0 0 bundle
[Sun Jan 10 00:05:35 2021] [ 27069] 996 27069 14829 1530 143360 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 10726] 996 10726 15611 3032 151552 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 43795] 996 43795 15497 2895 147456 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 39677] 0 39677 1126 17 49152 0 0 gitlab-logrotat
[Sun Jan 10 00:05:35 2021] [ 46365] 0 46365 1094 17 53248 0 0 sleep
[Sun Jan 10 00:05:35 2021] [ 4140] 996 4140 15522 3019 151552 0 0 postgres
[Sun Jan 10 00:05:35 2021] [ 27524] 0 27524 156033 358 143360 0 0 rsyslogd
[Sun Jan 10 00:05:35 2021] [ 23561] 0 23561 2785 98 57344 0 0 cron
[Sun Jan 10 00:05:35 2021] [ 23572] 0 23572 597 16 45056 0 0 sh
[Sun Jan 10 00:05:35 2021] [ 23573] 0 23573 2092 61 45056 0 0 ctl
[Sun Jan 10 00:05:35 2021] [ 23839] 0 23839 18370 1260 131072 0 0 php
[Sun Jan 10 00:05:35 2021] [ 24848] 0 24848 1358 42 53248 0 0 sshd
[Sun Jan 10 00:05:35 2021] Out of memory: Kill process 23562 (mysqld) score 436 or sacrifice child
[Sun Jan 10 00:05:35 2021] Killed process 23562 (mysqld) total-vm:36128600kB, anon-rss:28650288kB, file-rss:0kB, shmem-rss:0kB
[Sun Jan 10 00:05:37 2021] oom_reaper: reaped process 23562 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
답변1
설명이 있습니다사용 가능한 메모리가 있어도 시스템에서 메모리를 할당할 수 없습니다..
문제를 해결하는 방법에 관해서는 먼저 중지할 수 있는 메모리 누수나 사용되지 않는 프로세스가 있는지 확인하는 것이 좋습니다. 일단 메모리 관점에서 모든 것이 제대로 실행되고 있다는 것을 알게 되면 남은 것은 물리적 메모리를 늘리는 것뿐입니다. 그것은 불가능합니다. 스왑을 추가하면 일부 메모리가 확보되기 때문입니다(아마도 성능 저하가 있을 수 있음).
답변2
많은 사람들에게 물어봤지만 아직 좋은 해결책을 얻지 못했습니다.
나는 이 문제가 Linux 커널의 dentries/inode 캐시와 관련이 있다고 의심합니다. 사용 가능한 메모리가 부족하면 커널은 캐시 메모리를 회수하려고 시도하지 않습니다.
실행할 때마다 echo 2 > /proc/sys/vm/drop_caches메모리는 free약 8G가 됩니다. 그러다가 4G 정도까지 떨어지다가 수십분 정도 지속되다가 2G 정도까지 떨어지더니 결국에는 1G 이하, 때로는 400M 이하가 되기도 합니다.
사용 가능한 메모리 에만 2영향을 미치며 효과가 없습니다./proc/sys/vm/drop_caches1
echo 2 > /proc/sys/vm/drop_caches15분마다 실행되는 임시 솔루션(나쁘다고 생각하지만...)을 사용해야 했는데 ... 너무 안타깝습니다.