Azure / Azure/ip-masq-agent-v2

ip-masq-agent OOMKill'd in clusters with large amounts of iptables records

Open
#40 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
14
Forks
24
PR merge metrics
No merged PRs in 30d

Description

## Repro

- Create a default AKS cluster with kube-proxy in iptables mode
- Create a deployment of busybox pods with 50 replicas `kubectl create deployment busybox --image busybox --replicas 50`
- Run this gist to create 100 services each with 500 backend ports (50,000 backend port rules) https://gist.github.com/tyler-lloyd/5f6260b7ce6089fe944c35194cb841f7
- Observe ip-masq-agent getting OOMKill'd

---

It's unclear if ip-masq-agent is actually running out of memory or if it is getting killed by the host when trying to sync its iptables rules.

Error seen
```
F0701 15:14:36.026972 1 ip-masq-agent.go:156] the daemon encountered an error: error syncing masquerade rules: error creating chain "IP-MASQ-AGENT": exit status 4: Another app is currently holding the xtables lock; still 4s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still 3s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still 2s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still 1s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still 0s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock. Stopped waiting after 5s.
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0x1)
/go/pkg/mod/k8s.io/klog/v2@v2.40.1/klog.go:1140 +0x8a
k8s.io/klog/v2.(*loggingT).output(0xd40e40, 0x3, 0x0, 0xc0000a85b0, 0x0, {0xa918ee, 0x1}, 0xc0001bc140, 0x0)
/go/pkg/mod/k8s.io/klog/v2@v2.40.1/klog.go:1088 +0x66f
k8s.io/klog/v2.(*loggingT).printf(0x60, 0x8ae02108, 0x0, {0x0, 0x0}, {0x9410c5, 0x23}, {0xc0001bc140, 0x1, 0x1})
/go/pkg/mod/k8s.io/klog/v2@v2.40.1/klog.go:753 +0x1c5
k8s.io/klog/v2.Fatalf(...)
/go/pkg/mod/k8s.io/klog/v2@v2.40.1/klog.go:1642
main.main()
/src/cmd/ip-masq-agent-v2/ip-masq-agent.go:156 +0x408

goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
/go/pkg/mod/k8s.io/klog/v2@v2.40.1/klog.go:1283 +0x6a
created by k8s.io/klog/v2.init.0
/go/pkg/mod/k8s.io/klog/v2@v2.40.1/klog.go:420 +0xfb
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the AKS setup with 50 BusyBox replicas and the linked gist's 50,000 backend port rules, then observe ip-masq-agent memory and termination behavior. Start at cmd/ip-masq-agent-v2/ip-masq-agent.go:156 and inspect the iptables synchronization path around the reported xtables-lock failure. Done means the OOMKill or host termination cause is established and the agent handles this workload without failing.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go, kubernetes
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.