Stream container stats instead of per-cycle polling
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
Parent epic: #11216
## Main idea
`src/ai/backend/agent/docker/intrinsic.py:134–169` (`fetch_api_stats`) calls `container.stats(stream=False)` on every collection cycle, with a per-container 2 s timeout (`_CONTAINER_STAT_TIMEOUT`, line 73). `CPUPlugin` (line 301) and `MemoryPlugin` (line 791) both go through this path when `StatModes.DOCKER` is selected, and linuxkit hosts are forced onto it (line 370).
Two problems:
1. Every cycle pays a full HTTP round-trip to dockerd per container — typically 100–500 ms.
1. A single unresponsive container consumes the full 2 s timeout before the gather completes, delaying every other metric on the same cycle.
Switch to a long-lived streamed reader per container using `container.stats(stream=True)`, keep the latest sample in memory, and have the stat collector read from the in-memory cache instead of making new API calls. Start/stop the stream alongside the container's lifecycle.
## Alternative ideas
- Keep per-cycle polling but parallelize with a hard global deadline instead of per-container timeout. Smaller win, doesn't fix the round-trip cost.
- Do nothing here and rely on the sysfs-first path (tracked separately) on Linux; stream only on hosts where sysfs isn't available (linuxkit, Docker Desktop). This is probably the pragmatic order: land sysfs-first first, then stream only for the remaining fallback population.
## Anything else?
Needs care around stream lifecycle: leaked readers = leaked connections against dockerd. Should be wired to the same container-lifecycle hooks used for teardown. Also consider backpressure — dockerd emits stats every ~1 s, which is likely faster than needed; may want to downsample.
JIRA Issue: BA-5859
Contributor guide
Assessment
This issue has not been assessed yet.