lablup / lablup/backend.ai

Surface the cause of abnormal session termination (starting with OOM) in session status, not only container logs

Open
#13,463 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Problem

When a session container is OOM-killed, the session records `status_info = "self-terminated"` — the exact same value a container that exits normally on its own receives. Nothing distinguishes the two after the fact.

The only place the OOM is visible is the container log, which prints `Out-of-memory detected!` followed by the killed process tree. That log is retained — the agent stores it on `kernels.container_log` and `GET /session/{name}/logs` serves it after the session is dead. But reading it means already knowing which session to open, then scanning a per-session blob: it is not filterable across sessions and not aggregatable. It is also truncated from the beginning past `container-logs.max-length` (default 10 MiB), so a verbose workload — a training loop printing progress, exactly the kind of job that runs out of memory — can push the OOM line out of the retained window.

This is not a new observation. It was scoped once and half-delivered:

- GitHub #265 "Track 'oom killed'" (2022-07) asked for an `oom-killed` status info in addition to `self-terminated`, and raised the `State.OOMKilled` question directly.
- GitHub #629 attempted it. The review rejected the PAUSE/UNPAUSE-on-OOM strategy the PR had bundled in, but explicitly kept the rest in scope: "let's just make the agent to recognize OOM events and report them to the manager, and make the manager to store those as `status_info` and `status_data`."
- GitHub #1373 (merged 2023-07) delivered the container-log half only — it touched no manager-side files — and #265 was closed as resolved by it.

So the manager-side half was requested by the maintainer, agreed, and then never implemented.

## Why this matters now

GitHub #9806 / #9808 propose auto-restarting inference containers on crash, and name OOM as the motivating case. Once that lands, an OOM-killed replica comes back silently. Without a recorded cause, a crash-looping deployment surfaces only a rising retry count with nothing explaining it — the automation makes the diagnosis harder, not easier.

GitHub #5722 (buffer-memory reservation) is aimed at reducing OOM frequency and names "explainability to the customers/admins" as a goal. That goal needs this data to exist.

## What we want to be visible

1. **The cause is distinguishable.** An operator can tell an OOM kill apart from a clean self-exit, an external kill, and a crash. Today all four collapse into `self-terminated`.
1. **The cause is a structured field, not prose inside a log blob.** It can be read without opening and scanning a per-session log. This also removes the truncation risk: `container-logs.max-length` (default 10 MiB) truncates from the beginning, so a verbose workload can evict the OOM line from the retained log.
1. **It is visible at list level, not only in a detail view.** Scanning a list of terminated sessions should make the abnormal ones stand out. A cause that only appears after opening a per-session modal does not meet the need.
1. **Memory context is available where the cause is.** When an OOM is reported, knowing the limit and the observed usage at kill time is the difference between "raise the limit" and "fix the workload". Without it the report says something died but not what to do next.
1. **Which kernel is identified.** In a multi-node or multi-kernel session, which kernel hit the condition — not just that the session did.
1. **OOM that did not kill the session is still recorded.** The cgroup OOM killer can take a child process while the session keeps running. That is a real signal (the workload is at its ceiling) and today it is only ever a log line. It should be distinguishable from an OOM that actually ended the session.
1. **Consistent across session types.** Interactive, batch, and inference sessions report the cause the same way. An operator should not need to know the session type to know where to look.
1. **Machine-queryable.** "How many sessions died of OOM this week", "which images OOM most often", "is this deployment crash-looping on memory" should be answerable by query, not by grepping logs.

## Success criteria

- [ ] A terminated session whose container was OOM-killed is distinguishable from one that exited normally, without opening container logs
- [ ] The cause is still available when the container log has been truncated past `container-logs.max-length`
- [ ] Abnormal terminations are identifiable while scanning a list of sessions
- [ ] The report carries enough memory context to decide between raising the limit and fixing the workload
- [ ] In a multi-kernel session, the specific kernel is identified
- [ ] An OOM event that did not terminate the session is recorded and is distinguishable from one that did
- [ ] Interactive, batch, and inference sessions all report through the same path
- [ ] The data supports aggregate queries (count by cause, by image, by deployment)

## Notes / adjacent

- Only the Docker agent backend currently observes OOM at all; the Kubernetes backend has no equivalent path. Scoping the first delivery to one backend is fine, but the gap should be recorded rather than silently inherited.
- The in-progress containerd agent work subscribes to `/tasks/oom` and forwards it the same way the Docker backend does — i.e. to the container log only. Whatever shape this request takes should be reflected there too, or the same gap ships again in the new architecture.
- Separately, the WebUI status-info color map covers only 7 values and one of its keys (`creation-failed`) does not match any value the backend emits (`failed-to-create`). Most abnormal causes therefore render in the same neutral style as normal ones, and `self-terminated` renders green. This is a display-side gap that will limit the visible benefit of this request; worth filing on its own.

JIRA Issue: BA-7186

Contributor guide

Open the contributing guide

Research direction

Start at the Docker agent's existing OOM/container-log path and the manager persistence behind GET /session/{name}/logs; compare how terminated-session data appears in list responses. Done means an OOM event is structured, queryable, visible at list level, includes memory and kernel context, and is handled consistently across session types and agent backends.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
api, backend, distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.