Share a single DockerStatsStreamer across CPU/Memory intrinsic plugins
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
Parent epic: #11216
Follow-up to: #11224
## Main idea
After #11224, `CPUPlugin` and `MemoryPlugin` each own their own `DockerStatsStreamer`, and each constructs its own `aiodocker.Docker()` client in `init()`. That means every running container opens **two** persistent Docker stats streams across **two** separate `aiohttp.ClientSession` instances.
Aiohttp's default `TCPConnector(limit_per_host=30)` means at roughly 50+ containers per agent you start queueing stream setups and stacking backoff retries. Memory and connector overhead are also double what they need to be.
This becomes even more obviously worth refactoring once #11223 (sysfs-first CPU/memory stats) lands, because CPU/Memory `api_impl` paths will fall through to Docker only as a fallback — the persistent stream is really only needed for network / blkio fields neither plugin consumes today.
## Design
- Own a single `DockerStatsStreamer` on `DockerAgent` (or `AbstractAgent`, if the abstraction generalizes to kubernetes).
- Both `CPUPlugin` and `MemoryPlugin` consume via a shared reference (set during plugin init, or accessed through the existing `computer_ctx`).
- Lifecycle (`start` / `stop`) still hangs off `_handle_start_event` / `_handle_clean_event`; the agent just dispatches to a single streamer instead of each plugin.
- Post-#11223: consider moving the streamer out of the intrinsic plugins entirely and into a network/IO consumer, since that's what actually needs the stream.
## Dependencies
- Best sequenced AFTER #11223 (sysfs-first), since that clarifies what fields the stream still needs to deliver.
## Scope
- Refactor ownership of `DockerStatsStreamer`.
- Unify the two streamers into one instance per agent.
- Update `notify_container_started/destroyed` dispatch path accordingly.
- No behavioral change for operators — this is purely a resource-consumption fix.
## Out of scope
- Changing the streaming protocol or cadence.
- Kubernetes agent (`src/ai/backend/agent/kubernetes/`) — if it's affected, evaluate in parallel.
JIRA Issue: BA-5861
Contributor guide
Assessment
This issue has not been assessed yet.