kubernetes / kubernetes/kubernetes
kubelet should track tcp_mem stats also along with cpu/ram/disk
- Dominant language
- Go
- Stars
- 128k
- Forks
- 44.6k
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 208
Description
/kind feature
/sig node
**What happened**:
A program started leaking TCP memory, which filled up the node's TCP stack memory. The network performance on the node degraded and connections to pods running on the node either times out or will hang for a long time.
Node's `dmesg` had lines mentioning `TCP: out of memory -- consider tuning tcp_mem`
Further reading and investigation reveals that this could happen when TCP stack runs out of memory pages allocated by kernel or when there are lot of orphaned/open sockets.
TCP stack limits: max 86514
```bash
$ cat /proc/sys/net/ipv4/tcp_mem
43257 57676 86514
# min pressure max
```
Usage when issue happened: mem 87916
```
$ cat /proc/net/sockstat
sockets: used 1386
TCP: inuse 24 orphan 0 tw 58 alloc 863 mem 87916
UDP: inuse 3 mem 3
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
```
kubelet posts node status as ready.
**What you expected to happen**:
kubelet should say node is not ready.
It would be great if `kubelet` could track the `tcp_mem` stats also along with CPU/RAM/disk as network is also an important factor. If `tcp_mem` limit is hit, for some reason, the node is not usable. Notifying the user that node has some issue can help debugging and further identifying the cause.
**How to reproduce it (as minimally and precisely as possible)**:
1. Create a GKE cluster
2. Label a node to carry out tests
```bash
$ kubectl label node node=leak-test
```
3. Create an nginx deployment with Loadbalancer, with can serve a large file
```bash
$ kubectl create -f https://raw.githubusercontent.com/shahidhk/k8s-tcp-mem-leak/master/nginx.yaml
```
4. Check if you can download the large file
```bash
$ curl -o large-file /large-file
```
5. Create a deployment that can fill up the TCP stack memory
```bash
$ kubectl create -f https://raw.githubusercontent.com/shahidhk/k8s-tcp-mem-leak/master/leak-repro.yaml
```
6. SSH into the node and observe `cat /proc/sys/net/ipv4/tcp_mem` and `cat /proc/net/sockstat` and scale the deployment until the current mem exceeds the limit
7. Try downloading the large file again. It will either become very slow or will not happen at all
**Anything else we need to know?**:
This is more of a feature request for kubelet rather than a bug. TCP mem can get filled if the node is running a lot of TCP heavy workloads, need not necessarily be a leak. Since kubelet is ultimately responsible for reporting node's health, network should also be a parameter.
**Environment**:
- Kubernetes version (use `kubectl version`): `v1.9.6-gke.0`
- Cloud provider or hardware configuration: GKE, 1 node, n1-standard-1
- OS (e.g. from /etc/os-release): Container-Optimized OS
- Kernel (e.g. `uname -a`): 4.4.111+
- Install tools: -
- Others: -
Contributor guide
Research direction
The issue names no source files, tests, or entry points. Start by locating kubelet's node health and status reporting, then compare how CPU, memory, and disk conditions are handled with the proposed tcp_mem signal. Done means reaching a reviewed design and implementation plan for marking a node unhealthy when its TCP memory limit is exceeded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- infrastructure, operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100