.png)
나는Debian GNU/Linux 9.5(확장)커널 포함:4.9.0-7-amd64.
제가 겪고 있는 메모리 소비 문제의 원인이 FluentD 데몬으로 로그를 보내는 인앱 메커니즘이라는 것을 발견하고 TCP 메모리 사용량을 알아보려고 했습니다.
다음 출력에 따르면
/proc/net/sockstat:
sockets: used 779
TCP: inuse 23 orphan 0 tw 145 alloc 177 mem 4451
UDP: inuse 5 mem 2
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
memmetric은 TCP 메모리에서 사용되는 페이지 수(4K)입니다. 따라서 TCP 메모리 사용량은 다음과 같습니다.4451 * 4 = 17804k
ss -atmp:
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:ssh *:* users:(("sshd",pid=559,fd=3))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d712)
LISTEN 0 4096 127.0.0.1:8125 *:* users:(("netdata",pid=21419,fd=33))
skmem:(r0,rb33554432,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 4096 *:19999 *:* users:(("netdata",pid=21419,fd=4))
skmem:(r0,rb33554432,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 32768 *:3999 *:* users:(("protokube",pid=3504,fd=9))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 32768 127.0.0.1:19365 *:* users:(("kubelet",pid=2607,fd=10))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 32768 127.0.0.1:10248 *:* users:(("kubelet",pid=2607,fd=29))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 32768 127.0.0.1:10249 *:* users:(("kube-proxy",pid=3250,fd=10))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 128 *:sunrpc *:* users:(("rpcbind",pid=232,fd=8))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
ESTAB 0 0 172.18.25.47:ssh 46.198.221.224:35084 users:(("sshd",pid=20049,fd=3),("sshd",pid=20042,fd=3))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
TIME-WAIT 0 0 100.96.18.1:48226 100.96.18.110:3006
ESTAB 0 0 172.18.25.47:62641 172.18.18.165:3999 users:(("protokube",pid=3504,fd=11))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15390)
ESTAB 0 0 172.18.25.47:3999 172.18.63.198:46453 users:(("protokube",pid=3504,fd=17))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
SYN-SENT 0 1 172.18.25.47:28870 172.18.23.194:4000 users:(("protokube",pid=3504,fd=3))
skmem:(r0,rb12582912,t1280,tb12582912,f2816,w1280,o0,bl0,d0)
TIME-WAIT 0 0 100.96.18.1:34744 100.96.18.108:3008
ESTAB 0 0 172.18.25.47:3999 172.18.18.165:23733 users:(("protokube",pid=3504,fd=8))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
TIME-WAIT 0 0 100.96.18.1:12992 100.96.18.105:3007
TIME-WAIT 0 0 100.96.18.1:48198 100.96.18.110:3006
TIME-WAIT 0 0 100.96.18.1:63502 100.96.18.102:8001
ESTAB 0 0 127.0.0.1:10249 127.0.0.1:53868 users:(("kube-proxy",pid=3250,fd=5))
skmem:(r0,rb12582912,t0,tb12582912,f4096,w0,o0,bl0,d0)
TIME-WAIT 0 0 100.96.18.1:58032 100.96.18.101:3000
TIME-WAIT 0 0 100.96.18.1:17158 100.96.18.104:8000
ESTAB 0 0 172.18.25.47:38474 172.18.18.165:https users:(("kubelet",pid=2607,fd=38))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d112)
TIME-WAIT 0 0 100.96.18.1:17308 100.96.18.104:8000
ESTAB 0 0 127.0.0.1:32888 127.0.0.1:10255 users:(("go.d.plugin",pid=21570,fd=8))
skmem:(r0,rb12582912,t0,tb12582912,f20480,w0,o0,bl0,d3)
TIME-WAIT 0 0 100.96.18.1:57738 100.96.18.101:3000
TIME-WAIT 0 0 100.96.18.1:23650 100.96.18.97:3004
TIME-WAIT 0 0 100.96.18.1:34518 100.96.18.103:3001
ESTAB 0 0 127.0.0.1:53868 127.0.0.1:10249 users:(("go.d.plugin",pid=21570,fd=6))
skmem:(r0,rb12582912,t0,tb12582912,f8192,w0,o0,bl0,d1)
TIME-WAIT 0 0 100.96.18.1:23000 100.96.18.98:3002
ESTAB 0 0 172.18.25.47:38498 172.18.18.165:https users:(("kube-proxy",pid=3250,fd=7))
skmem:(r0,rb12582912,t0,tb12582912,f8192,w0,o0,bl0,d0)
TIME-WAIT 0 0 100.96.18.1:26430 100.96.18.100:3005
TIME-WAIT 0 0 100.96.18.1:34882 100.96.18.103:3001
ESTAB 0 0 172.18.25.47:3999 172.18.44.34:57033 users:(("protokube",pid=3504,fd=14))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
ESTAB 0 0 172.18.25.47:3999 172.18.25.148:60423 users:(("protokube",pid=3504,fd=18))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
ESTAB 0 0 172.18.25.47:61568 35.196.244.138:https users:(("netdata",pid=21419,fd=70))
skmem:(r0,rb12582912,t0,tb262176,f0,w0,o0,bl0,d0)
TIME-WAIT 0 0 100.96.18.1:13154 100.96.18.105:3007
ESTAB 0 0 172.18.25.47:54289 172.18.30.39:3999 users:(("protokube",pid=3504,fd=12))
skmem:(r0,rb12582912,t0,tb12582912,f4096,w0,o0,bl0,d15392)
TIME-WAIT 0 0 100.96.18.1:34718 100.96.18.108:3008
TIME-WAIT 0 0 100.96.18.1:24078 100.96.18.97:3004
LISTEN 0 128 :::ssh :::* users:(("sshd",pid=559,fd=4))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 4096 :::19999 :::* users:(("netdata",pid=21419,fd=5))
skmem:(r0,rb33554432,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 32768 :::4000 :::* users:(("protokube",pid=3504,fd=5))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 32768 :::32003 :::* users:(("kube-proxy",pid=3250,fd=13))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 32768 :::31719 :::* users:(("kube-proxy",pid=3250,fd=12))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 32768 :::10250 :::* users:(("kubelet",pid=2607,fd=24))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d23)
LISTEN 0 32768 :::9100 :::* users:(("node_exporter",pid=11027,fd=3))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 32768 :::31532 :::* users:(("kube-proxy",pid=3250,fd=11))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 32768 :::30892 :::* users:(("kube-proxy",pid=3250,fd=9))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 32768 :::10255 :::* users:(("kubelet",pid=2607,fd=26))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 128 :::sunrpc :::* users:(("rpcbind",pid=232,fd=11))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
LISTEN 0 32768 :::10256 :::* users:(("kube-proxy",pid=3250,fd=8))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d0)
TIME-WAIT 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.47:13492
ESTAB 0 0 ::ffff:172.18.25.47:10250 ::ffff:172.18.25.148:55670 users:(("kubelet",pid=2607,fd=40))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15400)
TIME-WAIT 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.47:13096
TIME-WAIT 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.47:13384
ESTAB 0 0 ::ffff:172.18.25.47:10250 ::ffff:172.18.44.34:49454 users:(("kubelet",pid=2607,fd=59))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d7698)
TIME-WAIT 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.47:13200
TIME-WAIT 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.47:13502
TIME-WAIT 0 0 ::ffff:172.18.25.47:4000 ::ffff:172.18.63.198:25438
TIME-WAIT 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.47:13586
TIME-WAIT 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.47:13298
ESTAB 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.148:45776 users:(("node_exporter",pid=11027,fd=7))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15419)
TIME-WAIT 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.47:13292
ESTAB 0 0 ::ffff:127.0.0.1:10255 ::ffff:127.0.0.1:32888 users:(("kubelet",pid=2607,fd=5))
skmem:(r0,rb12582912,t0,tb12582912,f4096,w0,o0,bl0,d0)
TIME-WAIT 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.47:13206
ESTAB 0 0 ::ffff:172.18.25.47:10250 ::ffff:172.18.18.165:33482 users:(("kubelet",pid=2607,fd=32))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d7707)
TIME-WAIT 0 0 ::ffff:172.18.25.47:4000 ::ffff:172.18.30.39:45200
TIME-WAIT 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.47:13594
TIME-WAIT 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.47:13390
TIME-WAIT 0 0 ::ffff:172.18.25.47:9100 ::ffff:172.18.25.47:13090
ESTAB 0 0 ::ffff:172.18.25.47:10250 ::ffff:172.18.25.148:55590 users:(("kubelet",pid=2607,fd=41))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15418)
ESTAB 0 0 ::ffff:172.18.25.47:10250 ::ffff:172.18.25.148:55536 users:(("kubelet",pid=2607,fd=11))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15401)
ESTAB 0 0 ::ffff:172.18.25.47:10250 ::ffff:172.18.25.148:55762 users:(("kubelet",pid=2607,fd=43))
skmem:(r0,rb12582912,t0,tb12582912,f0,w0,o0,bl0,d15407)
ss매뉴얼 에 따르면 :
skmem:(r<rmem_alloc>,rb<rcv_buf>,t<wmem_alloc>,tb<snd_buf>,
f<fwd_alloc>,w<wmem_queued>,o<opt_mem>,
bl<back_log>,d<sock_drop>)
<rmem_alloc>
the memory allocated for receiving packet
<rcv_buf>
the total memory can be allocated for receiving packet
<wmem_alloc>
the memory used for sending packet (which has been sent
to layer 3)
<snd_buf>
the total memory can be allocated for sending packet
<fwd_alloc>
the memory allocated by the socket as cache, but not used
for receiving/sending packet yet. If need memory to
send/receive packet, the memory in this cache will be
used before allocate additional memory.
<wmem_queued>
The memory allocated for sending packet (which has not
been sent to layer 3)
<ropt_mem>
The memory used for storing socket option, e.g., the key
for TCP MD5 signature
<back_log>
The memory used for the sk backlog queue. On a process
context, if the process is receiving packet, and a new
packet is received, it will be put into the sk backlog
queue, so it can be received by the process immediately
<sock_drop>
the number of packets dropped before they are de-multi‐
plexed into the socket
skmem합계를 제외한 rb모든 값을 더합니다.tb (할당할 수 있는 최대 금액이기 때문입니다)삭제된 패킷 의 경우 d값에 매우 가까운 값을 얻어야 합니다 /proc/net/sockstat. 그러나 내가 얻는 가치는53,00017804k와는 거리가 멀다.
내 논리가 맞나요? 그렇다면 내가 여기서 무엇을 놓치고 있는 걸까요?
답변1
몇번의 검색 끝에 마침내 결론을 내렸습니다.
TCP 메모리 사용량 계산 방법에 대한 나의 이해는 정확합니다.
각 소켓에 대해 추가socket_memory = rmem_alloc + wmem_alloc + fwd_alloc + wmem_queued + opt_mem + back_log ( , r, t, f, w의 필드blo스킴)
위에서 캡처한 총 소켓 메모리가 합산되지 않는 이유는 많은 연결이 도커 컨테이너 내부에서 실행되고 있기 때문입니다.기본 시스템 출력에는 표시되지 않지만 ss/proc/net/sockstat의 커널 출력에는 표시됩니다..
자세한 내용은 다음 유용한 stackoverflow 질문을 참조하세요.https://stackoverflow.com/questions/37171909/when-using-docker-built-connections-dont-appear-in-netstat
이것은 차이점을 설명합니다. 호스트 전용 프로세스의 경우 메모리 합계가 일치합니다.