kubernetes / kubernetes/kubernetes

kubelet should track tcp_mem stats also along with cpu/ram/disk

Open
#62,334 50 comments 57 reactions 0 assignees View on GitHub
kind/feature priority/backlog sig/network sig/node triage/accepted
Dominant language
Go
Stars
128k
Forks
44.6k
Avg merge
5d 17h
Merged PRs (30d)
208

Description

/kind feature
/sig node

**What happened**:

A program started leaking TCP memory, which filled up the node's TCP stack memory. The network performance on the node degraded and connections to pods running on the node either times out or will hang for a long time.

Node's `dmesg` had lines mentioning `TCP: out of memory -- consider tuning tcp_mem`

Further reading and investigation reveals that this could happen when TCP stack runs out of memory pages allocated by kernel or when there are lot of orphaned/open sockets.

TCP stack limits: max 86514

```bash
$ cat /proc/sys/net/ipv4/tcp_mem
43257 57676 86514
# min pressure max
```

Usage when issue happened: mem 87916

```
$ cat /proc/net/sockstat
sockets: used 1386
TCP: inuse 24 orphan 0 tw 58 alloc 863 mem 87916
UDP: inuse 3 mem 3
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0
```

kubelet posts node status as ready.

**What you expected to happen**:

kubelet should say node is not ready.

It would be great if `kubelet` could track the `tcp_mem` stats also along with CPU/RAM/disk as network is also an important factor. If `tcp_mem` limit is hit, for some reason, the node is not usable. Notifying the user that node has some issue can help debugging and further identifying the cause.

**How to reproduce it (as minimally and precisely as possible)**:

1. Create a GKE cluster
2. Label a node to carry out tests
```bash
$ kubectl label node node=leak-test
```
3. Create an nginx deployment with Loadbalancer, with can serve a large file
```bash
$ kubectl create -f https://raw.githubusercontent.com/shahidhk/k8s-tcp-mem-leak/master/nginx.yaml
```
4. Check if you can download the large file
```bash
$ curl -o large-file /large-file
```
5. Create a deployment that can fill up the TCP stack memory
```bash
$ kubectl create -f https://raw.githubusercontent.com/shahidhk/k8s-tcp-mem-leak/master/leak-repro.yaml
```
6. SSH into the node and observe `cat /proc/sys/net/ipv4/tcp_mem` and `cat /proc/net/sockstat` and scale the deployment until the current mem exceeds the limit
7. Try downloading the large file again. It will either become very slow or will not happen at all

**Anything else we need to know?**:

This is more of a feature request for kubelet rather than a bug. TCP mem can get filled if the node is running a lot of TCP heavy workloads, need not necessarily be a leak. Since kubelet is ultimately responsible for reporting node's health, network should also be a parameter.

**Environment**:
- Kubernetes version (use `kubectl version`): `v1.9.6-gke.0`
- Cloud provider or hardware configuration: GKE, 1 node, n1-standard-1
- OS (e.g. from /etc/os-release): Container-Optimized OS
- Kernel (e.g. `uname -a`): 4.4.111+
- Install tools: -
- Others: -

Contributor guide

Open the contributing guide

Research direction

The issue names no source files, tests, or entry points. Start by locating kubelet's node health and status reporting, then compare how CPU, memory, and disk conditions are handled with the proposed tcp_mem signal. Done means reaching a reviewed design and implementation plan for marking a node unhealthy when its TCP memory limit is exceeded.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux
Domain
infrastructure, operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.