Container and process metrics are not collected on Podman in the docker statistics mode
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
### Symptom
On a Podman-backed agent running with the default stats-type of "docker", not a single container metric is produced: the agent exposes zero backendai_container_utilization samples for a running kernel. Process metrics are lost as well, with this repeating every collection cycle:
```
ERROR ai.backend.agent.agent unhandled exception in collect_process_stat
Traceback (most recent call last):
File "src/ai/backend/agent/stats.py", line 885, in _get_processes
return_val.append(PID(int(proc[1])))
IndexError: list index out of range
```
### Cause 1 — the sample is discarded before it is read
fetch_api_stats() drops a statistics sample when either "read" or "preread" carries the zero timestamp. Podman always reports the zero value for "preread" on a non-streaming query, so every sample is discarded even though its counters are valid and advance between calls:
```
DOCKER read=...T10:24:58.893Z preread=...T10:24:57.886Z
PODMAN read=...T10:25:00.913Z preread=0001-01-01T00:00:00Z
cpu_usage.total_usage=6696482000 -> 6698306000 -> 6700029000
memory_stats.usage=100839424
```
The daemons themselves differ here; this is not introduced by the client library. Docker samples twice for a one-shot query and fills in the previous timestamp, Podman reads once. Only cumulative counters are read from the payload and the rates are derived from the agent's own sample history, so the discarded samples carry everything the caller needs.
### Cause 2 — the process table is indexed by fixed positions
_get_processes() reads column 1 of each row of the container process table. Both runtimes report the same Titles, but the rows differ in shape:
```
Titles (both) ['UID', 'PID', 'PPID', 'C', 'STIME', 'TTY', 'TIME', 'CMD']
DOCKER row ['ubuntu', '22849', '22795', '0', '02:54', '?', '00:00:00', '/app/...'] (8 fields)
PODMAN row ['ubuntu 7 1 0 09:04 pts/0 00:00:03 bai-krunner ...'] (1 field)
```
The resulting IndexError is not among the caught exception types, so it escapes and kills the whole collect_process_stat task rather than skipping one row. get_host_process_table() in the agent utilities carries the same fixed-position assumption for the PID and command columns.
### Impact
No container metrics (cpu, memory, io, net) and no process metrics on Podman. This is the only metric path available when the agent runs unprivileged, because the cgroup statistics mode is rejected unless the agent runs as root, so a rootless deployment has no metrics at all.
### Reproduced on
- Podman 4.9.3 rootless (Docker-compatible API), Ubuntu 24.04, x86_64
- Agent 26.8.0rc1 running unprivileged with the default stats-type
JIRA Issue: BA-7381
Contributor guide
Research direction
Start with fetch_api_stats() and _get_processes() in src/ai/backend/agent/stats.py, then inspect get_host_process_table() in the agent utilities. Compare the Docker and Podman payload shapes and process rows using the reproduced rootless setup. Done means Podman samples produce container metrics and process collection continues without the IndexError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- backend, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100