lablup / lablup/backend.ai

Investigate and fix permanent hang in Docker API containers/{id}/top call

Open
#9,322 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Background:

Analysis of BA-4651 identified that Docker API containers/{id}/top calls can permanently hang. While per-scope lock separation (BA-4653) prevents cascading failures, the affected scope (process stat) itself continues to fail.

Root Cause Analysis:

Docker's top API internally performs a ps-equivalent operation within the container's PID namespace, sequentially reading /proc/{pid}/stat, /proc/{pid}/wchan, etc. for each process.

If any process inside the container is in uninterruptible sleep state (D state), reading its /proc entries blocks indefinitely. D state typically occurs due to:

- Unresponsive NFS/FUSE/CIFS mounts inside the container

- Hung block device I/O (e.g., network storage failures)

- Unresolved kernel-level I/O waits

Since ps iterates all processes sequentially, a single D state process causes the entire top call to block indefinitely. Currently there is no per-call timeout for this operation (in contrast, Docker stats API in api_impl has a 2-second timeout). Only the outer STAT_COLLECTION_TIMEOUT (10 minutes) acts as a safety net.

Objective:

- Investigate the root cause of permanent hangs in Docker API containers/{id}/top calls

- Implement defensive/recovery logic based on findings (e.g., timeout with retry, skipping specific containers, Docker daemon health checks, etc.)

JIRA Issue: BA-4685

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.