Investigate and fix permanent hang in netstat_ns(sandbox_key) call
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
Background:
Analysis of BA-4651 identified that netstat_ns(sandbox_key) calls can permanently hang. While per-scope lock separation (BA-4653) prevents cascading failures, the affected scope (container stat) itself continues to fail.
Root Cause Analysis:
The netstat_ns() function spawns a separate process via ProcessPoolExecutor (or thread pool) that performs:
1. os.open(ns_path) — open the network namespace file
2. setns() — switch to the target namespace
3. psutil.net_io_counters(pernic=True) — read /proc/net/dev
4. setns() — switch back to the original namespace
Multiple scenarios can contribute to the hang:
1. Race condition during container teardown: When a container is shutting down and its network namespace is being torn down, the setns() call may block waiting on the kernel's namespace mutex. Simultaneous namespace destruction and entry can cause kernel-level deadlock.
2. Stale sandbox_key: The SandboxKey path (/var/run/docker/netns/...) obtained from container.show() may no longer be valid but not yet removed from the filesystem. In this case, os.open() succeeds but reading /proc/net/dev after setns() hangs.
3. ProcessPoolExecutor worker hang: When the above operations hang inside a worker process, the run_in_executor() await never completes. ProcessPoolExecutor has no per-task timeout mechanism.
Objective:
- Investigate the root cause of permanent hangs in netstat_ns(sandbox_key) calls
- Implement defensive/recovery logic based on findings (e.g., graceful skip when network namespace is inaccessible, stale sandbox_key detection, etc.)
JIRA Issue: BA-4686
Contributor guide
Assessment
This issue has not been assessed yet.