lablup / lablup/backend.ai-webui
Session post-mortem report with one-click adjusted retry
- Dominant language
- TypeScript
- Stars
- 133
- Forks
- 81
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 355
Description
## Problem
When a session dies (OOM, CUDA OOM, quota exceeded, image pull failure, scheduling failure), users must dig through container logs and status details to figure out why, then manually rebuild the launch config to try again. Most failure causes are already structured and detectable — we just never present them as a diagnosis.
## Proposal
Show a **post-mortem card** on terminated-with-error sessions (session detail drawer / status detail modal) that states the failure cause in plain language plus the evidence, and offer a **one-click retry** that re-launches the session with an adjusted spec.
### Phase 1 — rule-based (no AI required)
- Classify the failure from data the backend already provides:
- exit code 137 / kernel `status_info` + `status_data` (OOM, scheduling failure, image pull failure, quota)
- resource metrics history (e.g. "memory peaked at 98% of the limit")
- log-tail regex for common runtime errors (`torch.cuda.OutOfMemoryError`, disk full, etc.)
- Render a diagnosis card: cause, evidence, and a suggested fix (e.g. raise memory 16G → 24G).
- "Retry with suggested spec" button reuses the existing restore-from-history launch path (cf. FR-3394), pre-filling the launcher with the adjusted resource slots.
- Unknown causes fall back to a neutral "terminated with error" card with a link to the container logs.
### Phase 2 — optional LLM summary (follow-up, separate issue)
For stack traces no rule matches, summarize the log tail via an admin-configured inference endpoint served on the cluster itself (reusing the Chat endpoint infrastructure). Off by default; Phase 1 must be complete and useful without it.
## Touchpoints
- `SessionStatusDetailModal` / `SessionStatusTag` (`status_info` / `status_data` readers)
- `ContainerLogModal` log fetch path (log-tail sampling)
- Session detail drawer (card placement)
- Session launcher restore-from-history flow (pre-filled adjusted spec)
## Acceptance criteria
- An OOM-killed session shows an OOM diagnosis card with the measured memory evidence.
- The retry button opens the launcher pre-filled with the adjusted spec (adjustment visibly highlighted).
- Sessions with unrecognized failure causes degrade gracefully to the current behavior plus a log link.
- No LLM/inference dependency in Phase 1.
JIRA Issue: FR-3647
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing SessionStatusDetailModal and SessionStatusTag, including their status_info/status_data readers, then inspect ContainerLogModal's log fetch path and the session launcher restore-from-history flow. The session detail drawer is the proposed card location. Done means recognized failures show evidence and an adjusted retry, while unknown causes retain the current behavior with a log link and no Phase 1 LLM dependency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- frontend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100