Investigate and fix permanent hang in Docker API containers/{id}/top call
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
Background:
Analysis of BA-4651 identified that Docker API containers/{id}/top calls can permanently hang. While per-scope lock separation (BA-4653) prevents cascading failures, the affected scope (process stat) itself continues to fail.
Root Cause Analysis:
Docker's top API internally performs a ps-equivalent operation within the container's PID namespace, sequentially reading /proc/{pid}/stat, /proc/{pid}/wchan, etc. for each process.
If any process inside the container is in uninterruptible sleep state (D state), reading its /proc entries blocks indefinitely. D state typically occurs due to:
- Unresponsive NFS/FUSE/CIFS mounts inside the container
- Hung block device I/O (e.g., network storage failures)
- Unresolved kernel-level I/O waits
Since ps iterates all processes sequentially, a single D state process causes the entire top call to block indefinitely. Currently there is no per-call timeout for this operation (in contrast, Docker stats API in api_impl has a 2-second timeout). Only the outer STAT_COLLECTION_TIMEOUT (10 minutes) acts as a safety net.
Objective:
- Investigate the root cause of permanent hangs in Docker API containers/{id}/top calls
- Implement defensive/recovery logic based on findings (e.g., timeout with retry, skipping specific containers, Docker daemon health checks, etc.)
JIRA Issue: BA-4685
Contributor guide
Assessment
This issue has not been assessed yet.