apache / apache/cloudstack

iptables_check.py health check deadlocks when iptables output exceeds the pipe buffer, leaking processes until the VR exhausts memory and self-reboots (~9h cycle)

Open Beginner friendly
#13,735 3 comments 0 reactions 0 assignees View on GitHub
component:virtual-router Severity:Critical type:bug
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

##### ISSUE TYPE

* Bug Report

##### COMPONENT NAME

~~~
SystemVM / Virtual Router health checks (systemvm/debian/root/health_checks/iptables_check.py)
~~~

##### CLOUDSTACK VERSION

~~~
4.22.0.0 and 4.22.1.0 (management), systemvm template 4.22.0.
The affected code is unchanged on current main:
https://github.com/apache/cloudstack/blob/main/systemvm/debian/root/health_checks/iptables_check.py
~~~

##### CONFIGURATION

Redundant VPC virtual routers, KVM. Advanced zone. Default health check settings
(`router.health.checks.enabled=true`, `router.health.checks.advanced.interval=10`).
The affected VPC has ~200 port-forwarding/LB rules (iptables-save output: 77,029 bytes).

##### OS / ENVIRONMENT

KVM hosts on Rocky Linux 9.5/9.8 (qemu-kvm 9.0/10.1 — reproduced on both). Debian-based
system VM template, 512 MB RAM routers.

##### SUMMARY

`iptables_check.py` runs the fetch command with `Popen(cmd, shell=True, stdout=PIPE)` and
then calls `pout.wait()` **before** reading the pipe:

```python
fetchIpTableEntriesCmd = "iptables-save | grep " + destIp
pout = Popen(fetchIpTableEntriesCmd, shell=True, stdout=PIPE)
if pout.wait() != 0: # <-- deadlocks: wait() before the pipe is drained
...
ipTablesMatchingEntries = pout.communicate()[0].decode().strip().split('\n')
```

This is the deadlock explicitly warned about in the Python `subprocess` documentation:
when the command's output exceeds the OS pipe capacity (64 KiB on Linux), the child
blocks writing to the full pipe and the parent blocks forever in `wait()`.

On routers with large rule sets the check therefore hangs **on every run**. Because the
advanced health check fires every `router.health.checks.advanced.interval` (10 min
default), a new stuck process chain (`monitorServices.py advanced` -> `sh` ->
`iptables_check.py`) accumulates every interval — ~13 MB RSS each, roughly 85–90 MB/h.
On a 512 MB router, RAM and swap are exhausted after ~8.5–9 h of uptime. At that point
keepalived's `heartbeat` track script can no longer fork within its timeout
(`VRRP_Script(heartbeat) timed_out` -> `Entering FAULT STATE`), the router demotes,
wipes `/etc/keepalived/keepalived.conf`, and the guest reboots — taking the VPC down.

The impact is amplified on redundant VPCs: both routers of a pair are created together,
so their exhaustion clocks are synchronized and **both routers fail within minutes of
each other**, defeating the redundancy. We experienced a full-VPC outage roughly every
9 hours until the check was excluded.

##### STEPS TO REPRODUCE

~~~
1. Create a VPC with enough port-forwarding/LB rules that the health check's
"iptables-save | grep " output exceeds 64 KiB inside the router
(our affected router: iptables-save = 77,029 bytes total).
2. Leave router.health.checks.enabled=true with default advanced interval (10 min).
3. Inside the router, watch: ps -eo pid,ppid,etimes,rss,args | grep -E "monitorServices|iptables_check"
4. A new stuck chain appears every 600 s and never exits (parent and child both in
wchan do_wait; the shell child is blocked writing to the full pipe).
5. Watch "free -k": used memory climbs linearly; after ~8.5-9 h the router enters
keepalived FAULT, demotes/wipes, and reboots.
~~~

Observed accumulation (etimes exactly 600 s apart, none ever exiting):

~~~
90114 7551 /usr/bin/python /root/monitorServices.py advanced
90136 7551 /bin/sh -c ./health_checks/iptables_check.py advanced
90137 7551 /usr/bin/python ./health_checks/iptables_check.py advanced
92506 6952 /usr/bin/python /root/monitorServices.py advanced
... 6351 (repeats every 600s)
... 5751
... 5152
... 4552
~~~

Control group: three routers in other VPCs on the same platform with small rule sets
(iptables-save 1.2–9.3 KB) run the identical check to completion in <1 s and have
152-day uptimes. The only variable is iptables output size vs the 64 KiB pipe buffer.

##### EXPECTED RESULTS

~~~
The health check completes (or fails fast) regardless of iptables rule-set size.
~~~

##### ACTUAL RESULTS

~~~
The check hangs permanently on every run once the piped output exceeds 64 KiB.
Stuck process chains accumulate every advanced-check interval until the router
exhausts RAM+swap (~8.5-9 h on a 512 MB router), keepalived's track script starves,
the router demotes/wipes its keepalived config and reboots. Redundant pairs fail
near-simultaneously because their clocks are synchronized at creation, causing a
recurring full-VPC outage.
~~~

##### WORKAROUND

Set `router.health.checks.to.exclude = iptables_check.py` (global setting, dynamic).
Note: in our testing the value had to be the script filename **with** the `.py`
extension — the bare name `iptables_check` was silently ignored and the check kept
running (possibly a second, minor bug in how the exclusion list is matched).
Accumulated stuck processes must be killed manually inside affected routers.

##### SUGGESTED FIX

Do not call `wait()` while `stdout=PIPE` is unread. Use `communicate()` first and
check `returncode` afterwards, e.g.:

```python
pout = Popen(fetchIpTableEntriesCmd, shell=True, stdout=PIPE)
stdout, _ = pout.communicate()
if pout.returncode != 0:
...
ipTablesMatchingEntries = stdout.decode().strip().split('\n')
```

The same `Popen(stdout=PIPE)` + `wait()` pattern exists in other health check scripts
(e.g. `gateways_check.py`, `cpu_usage_check.py`). Their outputs are normally tiny so
they do not currently hang, but they share the same latent hazard and may be worth
fixing in the same pass.

Contributor guide

Open the contributing guide

Research direction

Start in systemvm/debian/root/health_checks/iptables_check.py and trace the Popen call used by the advanced check. Run it with an iptables-save | grep output larger than the pipe buffer, then verify the check completes and leaves no stuck process; compare gateways_check.py and cpu_usage_check.py for the same wait pattern if widening scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, python, shell
Domain
infrastructure, networking
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.