lablup / lablup/backend.ai

Agent: a single stuck (D-state) container stalls all process metric collection

Open
#12,645 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Objective

Make the agent per-container process metric collection resilient so that one unresponsive (D-state / un-killable) container cannot stall metric collection for all other healthy containers on the same agent.

## Background

In a production incident, session/process metrics stopped being reported for an agent. Investigation showed one container was stuck in D-state (uninterruptible sleep), likely caused by a backing storage (loop scratch filesystem on /data2) I/O failure. dockerd could not kill it: the delete API returned DockerError(500) "could not kill container: tried to kill container, but did not receive an exit event", and container log collection also timed out for that container. The scratch umount then failed (RuntimeError: umount failed), and the whole destroy/clean cycle kept retrying every ~60-90s.

The root cause of the container hang is external (storage I/O). The agent-side defect is the lack of isolation: StatContext.collect_per_container_process_stat (src/ai/backend/agent/stats.py) builds the PID map with a SEQUENTIAL loop over all containers, each calling the docker top API (_get_processes -> GET containers/{cid}/top) with NO per-call timeout. When docker top hangs on the stuck container (same reason logs/delete hang), the loop never advances, so containers after it never get collected. Because the actual measurement gather and the Redis write happen AFTER the loop, even already-iterated healthy containers get nothing written for that cycle.

The 5s timer uses TimerDelayPolicy.CANCEL, so each cycle cancels and restarts the hung task, but the fresh task hangs again on the same container. Net effect: process metrics stop for ALL containers on the agent as long as one container is stuck. Note: the container-stat path (collect_container_stat) is more resilient because it fans out per-container with asyncio.create_task + gather and applies per-container/per-plugin timeouts; the process-stat path should follow the same pattern.

## Scope

- Change _get_processes / collect_per_container_process_stat so the PID discovery runs per-container concurrently (asyncio.create_task + gather(return_exceptions=True)) instead of a sequential await loop.
- Add a per-container timeout around the docker top call; on timeout/error, skip that container and continue collecting the rest.
- Ensure the Redis write for healthy containers is not blocked by a single stuck container.

## Acceptance Criteria

- With one container whose docker top call hangs, process metrics for all other healthy containers are still collected and written to Redis every cycle.
- The docker top call for each container is bounded by a timeout; a stuck container is skipped for that cycle and logged, not fatal to the loop.
- A regression test simulates one hanging/slow container and asserts the others are still measured.

## Notes / Follow-ups

- Secondary risk (separate issue candidate): container-stat cgroup/proc reads (read_sysfs / Path.read_text) run synchronously on the event loop; if a read blocks uninterruptibly it can freeze the whole agent event loop, which asyncio.timeout cannot interrupt.

JIRA Issue: BA-6773

Contributor guide

Open the contributing guide

Research direction

Start in src/ai/backend/agent/stats.py at StatContext.collect_per_container_process_stat and _get_processes, then compare the existing concurrent container-stat collection path. Trace the Docker containers/{cid}/top call and its timeout handling, and locate the agent stats tests. Done means a hanging container is skipped and healthy containers still have process metrics written to Redis each cycle.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.