Azure / Azure/ip-masq-agent-v2
ip-masq-agent OOMKill'd in clusters with large amounts of iptables records
- Dominant language
- Go
- Stars
- 14
- Forks
- 24
- PR merge metrics
- No merged PRs in 30d
Description
## Repro
- Create a default AKS cluster with kube-proxy in iptables mode
- Create a deployment of busybox pods with 50 replicas `kubectl create deployment busybox --image busybox --replicas 50`
- Run this gist to create 100 services each with 500 backend ports (50,000 backend port rules) https://gist.github.com/tyler-lloyd/5f6260b7ce6089fe944c35194cb841f7
- Observe ip-masq-agent getting OOMKill'd
---
It's unclear if ip-masq-agent is actually running out of memory or if it is getting killed by the host when trying to sync its iptables rules.
Error seen
```
F0701 15:14:36.026972 1 ip-masq-agent.go:156] the daemon encountered an error: error syncing masquerade rules: error creating chain "IP-MASQ-AGENT": exit status 4: Another app is currently holding the xtables lock; still 4s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still 3s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still 2s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still 1s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still 0s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock. Stopped waiting after 5s.
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0x1)
/go/pkg/mod/k8s.io/klog/v2@v2.40.1/klog.go:1140 +0x8a
k8s.io/klog/v2.(*loggingT).output(0xd40e40, 0x3, 0x0, 0xc0000a85b0, 0x0, {0xa918ee, 0x1}, 0xc0001bc140, 0x0)
/go/pkg/mod/k8s.io/klog/v2@v2.40.1/klog.go:1088 +0x66f
k8s.io/klog/v2.(*loggingT).printf(0x60, 0x8ae02108, 0x0, {0x0, 0x0}, {0x9410c5, 0x23}, {0xc0001bc140, 0x1, 0x1})
/go/pkg/mod/k8s.io/klog/v2@v2.40.1/klog.go:753 +0x1c5
k8s.io/klog/v2.Fatalf(...)
/go/pkg/mod/k8s.io/klog/v2@v2.40.1/klog.go:1642
main.main()
/src/cmd/ip-masq-agent-v2/ip-masq-agent.go:156 +0x408
goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
/go/pkg/mod/k8s.io/klog/v2@v2.40.1/klog.go:1283 +0x6a
created by k8s.io/klog/v2.init.0
/go/pkg/mod/k8s.io/klog/v2@v2.40.1/klog.go:420 +0xfb
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the AKS setup with 50 BusyBox replicas and the linked gist's 50,000 backend port rules, then observe ip-masq-agent memory and termination behavior. Start at cmd/ip-masq-agent-v2/ip-masq-agent.go:156 and inspect the iptables synchronization path around the reported xtables-lock failure. Done means the OOMKill or host termination cause is established and the agent handles this workload without failing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go, kubernetes
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100