최근 FreeBSD 10.3-STABLE에서 FreeBSD 10.3-RELEASE-p21로 업그레이드된 FreeBSD 서버에서 mbuf 소모가 발생했습니다. 정상 작동 중에는 제한에 도달할 때까지 mbuf 사용량이 꾸준히 증가하는 것을 볼 수 있습니다 kern.ipc.nmbufs. 이 시점에서 시스템은 네트워크에서 응답하지 않게 되며(네트워크 액세스를 위한 mbuf 부족으로 인해) 콘솔에 다음이 표시됩니다.
cxl0: Interface stopped DISTRIBUTING, possible flapping
cxl1: Interface stopped DISTRIBUTING, possible flapping
[zone: mbuf] kern.ipc.nmbufs limit reached
[zone: mbuf] kern.ipc.nmbufs limit reached
이 시스템은 pf를 실행하고 패킷 필터, 라우터, 게이트웨이 및 DHCP/DNS 서버 역할을 합니다. 두 개의 Chelsio NIC가 있고 보조 CARP 마스터입니다. 보조 장치에는 동일한 하드웨어 및 소프트웨어 구성이 있으며 이 문제가 없습니다.
netstat -m이로 인해 가동 중지 시간이 발생할 수 있다는 점을 고려하여 출력을 표시 하고 mbufs in use접근 방식에 대해 경고하도록 Nagios/Check_MK를 설정했으며 kern.ipc.nmbufs재부팅할 때까지 mbuf 사용량이 꾸준히 선형으로 증가하는 것을 확인했습니다.
mbuf 클러스터 사용 중이 경우 아무 것도 변경되지 않으며 mbuf 클러스터 제한을 늘려도 효과가 없습니다.
이것은 제가 보기에 일종의 커널 버그인 것 같습니다. 추가 문제 해결에 대한 제안을 찾고 있거나 이 문제를 해결하는 데 도움이 필요합니다!
유용한(아마도) 정보:
netstat -m:
679270/3080/682350 mbufs in use (current/cache/total)
10243/1657/11900/985360 mbuf clusters in use (current/cache/total/max)
10243/1648 mbuf+clusters out of packet secondary zone in use (current/cache)
8128/482/8610/124025 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/36748 9k jumbo clusters in use (current/cache/total/max)
128/0/128/20670 16k jumbo clusters in use (current/cache/total/max)
224863K/6012K/230875K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
vmstat -z|grep -E '^ITEM|mbuf':
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
mbuf_packet: 256, 1587540, 10239, 1652,84058893, 0, 0
mbuf: 256, 1587540, 671533, 1206,914478880, 0, 0
mbuf_cluster: 2048, 985360, 11891, 9, 11891, 0, 0
mbuf_jumbo_page: 4096, 124025, 8128, 512,15011847, 0, 0
mbuf_jumbo_9k: 9216, 36748, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 20670, 128, 0, 128, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
vmstat -m:
Type InUse MemUse HighUse Requests Size(s)
NFSD lckfile 1 1K - 1 256
filedesc 103 383K - 1134731 16,32,128,2048,4096,8192,16384,65536
sigio 1 1K - 1 64
filecaps 0 0K - 973 64
kdtrace 292 59K - 1099386 64,256
kenv 121 13K - 125 16,32,64,128,8192
kqueue 14 22K - 5374 256,2048,8192
proc-args 54 5K - 578448 16,32,64,128,256
hhook 2 1K - 2 256
ithread 146 24K - 146 32,128,256
KTRACE 100 13K - 100 128
NFS fh 1 1K - 584 32
linker 207 1052K - 234 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
lockf 29 3K - 20042 64,128
loginclass 2 1K - 1192 64
devbuf 17205 36362K - 17523 16,32,64,128,256,512,1024,2048,4096,8192,65536
temp 149 51K - 1280113 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
ip6opt 5 2K - 6 256
ip6ndp 27 2K - 27 64,128
module 230 29K - 230 128
mtx_pool 2 16K - 2 8192
osd 3 1K - 5 16,32,64
pmchooks 1 1K - 1 128
pgrp 30 4K - 2222 128
session 29 4K - 2187 128
proc 2 32K - 2 16384
subproc 211 368K - 1099014 512,4096
cred 204 32K - 6025704 64,256
plimit 19 5K - 3985 256
uidinfo 9 5K - 11892 128,4096
NFSD session 1 1K - 1 1024
sysctl 0 0K - 63851 16,32,64
sysctloid 7196 365K - 7369 16,32,64,128
sysctltmp 0 0K - 17834 16,32,64,128
tidhash 1 32K - 1 32768
callout 5 2184K - 5
umtx 522 66K - 522 128
p1003.1b 1 1K - 1 16
SWAP 2 549K - 2 64
bus 802 86K - 6536 16,32,64,128,256,1024
bus-sc 57 1671K - 2431 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
newnfsmnt 1 1K - 1 1024
devstat 8 17K - 8 32,4096
eventhandler 116 10K - 116 64,128
kobj 124 496K - 296 4096
acpiintr 1 1K - 1 64
Per-cpu 1 1K - 1 32
acpica 14355 1420K - 216546 16,32,64,128,256,512,1024,2048,4096
pci_link 16 2K - 16 64,128
pfs_nodes 21 6K - 21 256
rman 316 37K - 716 16,32,128
sbuf 1 1K - 41375 16,32,64,128,256,512,1024,2048,4096,8192,16384
sglist 8 8K - 8 1024
GEOM 88 15K - 1871 16,32,64,128,256,512,1024,2048,8192,16384
acpipwr 5 1K - 5 64
taskqueue 43 7K - 43 16,32,256
Unitno 22 2K - 1208250 32,64
vmem 3 144K - 6 1024,4096,8192
ioctlops 0 0K - 185700 256,512,1024,2048,4096
select 89 12K - 89 128
iov 0 0K - 19808992 16,64,128,256,512,1024
msg 4 30K - 4 2048,4096,8192,16384
sem 4 106K - 4 2048,4096
shm 1 32K - 1 32768
tty 20 20K - 499 1024
pts 1 1K - 480 256
accf 2 1K - 2 64
mbuf_tag 0 0K - 291472282 32,64,128
shmfd 1 8K - 1 8192
soname 32 4K - 1210442 16,32,128
pcb 36 663K - 76872 16,32,64,128,1024,2048,8192
CAM CCB 0 0K - 182128 2048
acl 0 0K - 2 4096
vfscache 1 2048K - 1
cl_savebuf 0 0K - 480 64
vfs_hash 1 1024K - 1
vnodes 1 1K - 1 256
entropy 1026 65K - 49107 32,64,4096
mount 64 3K - 140 16,32,64,128,256
vnodemarker 0 0K - 4212 512
BPF 112 20504K - 131 16,64,128,512,4096
CAM path 11 1K - 63 32
ifnet 29 57K - 30 128,256,2048
ifaddr 315 105K - 315 32,64,128,256,512,2048,4096
ether_multi 232 13K - 282 16,32,64
clone 10 2K - 10 128
arpcom 23 1K - 23 16
gif 4 1K - 4 32,256
lltable 155 53K - 551 256,512
UART 6 5K - 6 16,1024
vlan 56 5K - 74 64,128
acpitask 1 16K - 1 16384
acpisem 110 14K - 110 128
raid_data 0 0K - 108 32,128,256
routetbl 516 136K - 101735 32,64,128,256,512
igmp 28 7K - 28 256
CARP 76 30K - 83 16,32,64,128,256,512,1024
ipid 2 24K - 2 8192,16384
in_mfilter 112 112K - 112 1024
in_multi 43 11K - 43 256
ip_moptions 224 35K - 224 64,256
CAM periph 7 2K - 19 16,32,64,128,256
acpidev 128 8K - 128 64
CAM queue 15 5K - 39 16,32,512
encap_export_host 4 4K - 4 1024
sctp_a_it 0 0K - 36 16
sctp_vrf 1 1K - 1 64
sctp_ifa 115 15K - 204 128
sctp_ifn 21 3K - 23 128
sctp_iter 0 0K - 36 256
hostcache 1 32K - 1 32768
syncache 1 64K - 1 65536
in6_mfilter 1 1K - 1 1024
in6_multi 15 2K - 15 32,256
ip6_moptions 2 1K - 2 32,256
CAM dev queue 6 1K - 6 64
kbdmux 6 22K - 6 16,512,1024,2048,16384
mld 26 4K - 26 128
LED 20 2K - 20 16,128
inpcbpolicy 365 12K - 119277 32
secasvar 7 2K - 214 256
sahead 10 3K - 10 256
ipsecpolicy 748 187K - 241562 256
ipsecrequest 18 3K - 72 128
ipsec-misc 56 2K - 1712 16,32,64
ipsec-saq 0 0K - 24 128
ipsec-reg 3 1K - 3 32
pfsync 2 2K - 893 32,256,1024
pf_temp 0 0K - 78 128
pf_hash 3 2880K - 3
pf_ifnet 36 11K - 9510 256,2048
pf_tag 7 1K - 7 128
pf_altq 5 2K - 125 256
pf_rule 964 904K - 17500 128,1024
pf_osfp 1130 115K - 28250 64,128
pf_table 49 98K - 948 2048
crypto 37 11K - 1072 64,128,256,512,1024
xform 7 1K - 1530156 16,32,64,128,256
rpc 12 20K - 304 64,128,512,1024,8192
audit_evclass 187 6K - 231 32
ufs_dirhash 93 18K - 93 16,32,64,128,256,512
ufs_quota 1 1024K - 1
ufs_mount 3 13K - 3 512,4096,8192
vm_pgdata 2 513K - 2 128
UMAHash 5 6K - 10 512,1024,2048
CAM SIM 6 2K - 6 256
CAM XPT 30 3K - 1850 16,32,64,128,256,512,1024,2048,65536
CAM DEV 9 18K - 16 2048
fpukern_ctx 3 6K - 3 2048
memdesc 1 4K - 1 4096
USB 23 33K - 24 16,128,256,512,1024,2048,4096
DEVFS3 136 34K - 2027 256
DEVFS1 108 54K - 594 512
apmdev 1 1K - 1 128
madt_table 0 0K - 1 4096
DEVFS_RULE 55 26K - 55 64,512
DEVFS 12 1K - 13 16,128
DEVFSP 22 2K - 167 64
io_apic 1 2K - 1 2048
isadev 8 1K - 8 128
MCA 15 2K - 15 32,128
msi 30 4K - 30 128
nexusdev 5 1K - 5 16
USBdev 21 8K - 21 32,64,128,256,512,1024,4096
NFSD V4client 1 1K - 1 256
cdev 5 2K - 5 256
cxgbe 41 956K - 44 128,256,512,1024,2048,4096,8192,16384
ipmi 0 0K - 20155 128,2048
htcp data 127 4K - 13675 32
aesni_data 3 3K - 3 1024
solaris 142 12302K - 3189 16,32,64,128,512,1024,8192
kstat_data 6 1K - 6 64
TCP 상태:
답변1
확실히 커널 버그처럼 보입니다. FreeBSD 11.1로 업그레이드해 볼 수 있나요?