lablup / lablup/backend.ai

Default to sysfs-first CPU/memory stats on Linux

Open
#11,220 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

Parent epic: #11216

## Main idea

The agent already has two stat-collection implementations: a sysfs direct-read path (fast, ~1–10 ms, reads `/sys/fs/cgroup/...` directly) and a Docker-API path (slow, 100–500 ms per container). Selection is driven by `StatModes.CGROUP` vs `StatModes.DOCKER` in `src/ai/backend/agent/docker/intrinsic.py:262–347` (CPU) and `:625–900` (memory).

Today the selection is a static config decision. On Linux native hosts (the primary deployment target), sysfs is always available and strictly faster, but the default selection logic does not prefer it. Linuxkit / Docker Desktop hosts (line 370) legitimately need the API path.

Change the default so Linux native hosts pick `CGROUP` mode automatically, fall back to `DOCKER` only when sysfs reads fail or the host is linuxkit/Docker Desktop. Keep the explicit config override.

Per-resource behavior to preserve:

- CPU: cgroup `cpu.stat` / `cpuacct.usage` read — already implemented for both cgroupv1 (line 275 match) and cgroupv2 (line 287).
- Memory: `memory.current`, `memory.stat` — already implemented for both versions (lines 661, 694).
- Network / block IO: keep using Docker API; these are not cleanly available from container cgroups and require netns entry which is costlier than the API call.

## Alternative ideas

- Remove the `DOCKER` mode entirely for CPU and memory. Cleaner, but loses the escape hatch for hosts where sysfs paths move (rootless, nested containers, unusual cgroup drivers).
- Add a runtime healthcheck on startup that benchmarks both paths and picks the winner. Overkill for what is essentially a Linux-native vs not decision.

## Anything else?

Pairs well with issue about streaming stats (#11219) — once sysfs is the default for CPU/memory, the streamed path only needs to cover network/IO and the linuxkit fallback, which keeps the stream connection count and complexity down.

JIRA Issue: BA-5860

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.