lablup / lablup/backend.ai-webui

Session post-mortem report with one-click adjusted retry

Open
#8,999 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
133
Forks
81
Avg merge
1d 12h
Merged PRs (30d)
355

Description

## Problem

When a session dies (OOM, CUDA OOM, quota exceeded, image pull failure, scheduling failure), users must dig through container logs and status details to figure out why, then manually rebuild the launch config to try again. Most failure causes are already structured and detectable — we just never present them as a diagnosis.

## Proposal

Show a **post-mortem card** on terminated-with-error sessions (session detail drawer / status detail modal) that states the failure cause in plain language plus the evidence, and offer a **one-click retry** that re-launches the session with an adjusted spec.

### Phase 1 — rule-based (no AI required)

- Classify the failure from data the backend already provides:
- exit code 137 / kernel `status_info` + `status_data` (OOM, scheduling failure, image pull failure, quota)
- resource metrics history (e.g. "memory peaked at 98% of the limit")
- log-tail regex for common runtime errors (`torch.cuda.OutOfMemoryError`, disk full, etc.)
- Render a diagnosis card: cause, evidence, and a suggested fix (e.g. raise memory 16G → 24G).
- "Retry with suggested spec" button reuses the existing restore-from-history launch path (cf. FR-3394), pre-filling the launcher with the adjusted resource slots.
- Unknown causes fall back to a neutral "terminated with error" card with a link to the container logs.

### Phase 2 — optional LLM summary (follow-up, separate issue)

For stack traces no rule matches, summarize the log tail via an admin-configured inference endpoint served on the cluster itself (reusing the Chat endpoint infrastructure). Off by default; Phase 1 must be complete and useful without it.

## Touchpoints

- `SessionStatusDetailModal` / `SessionStatusTag` (`status_info` / `status_data` readers)
- `ContainerLogModal` log fetch path (log-tail sampling)
- Session detail drawer (card placement)
- Session launcher restore-from-history flow (pre-filled adjusted spec)

## Acceptance criteria

- An OOM-killed session shows an OOM diagnosis card with the measured memory evidence.
- The retry button opens the launcher pre-filled with the adjusted spec (adjustment visibly highlighted).
- Sessions with unrecognized failure causes degrade gracefully to the current behavior plus a log link.
- No LLM/inference dependency in Phase 1.

JIRA Issue: FR-3647

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing SessionStatusDetailModal and SessionStatusTag, including their status_info/status_data readers, then inspect ContainerLogModal's log fetch path and the session launcher restore-from-history flow. The session detail drawer is the proposed card location. Done means recognized failures show evidence and an adjusted retry, while unknown causes retain the current behavior with a log link and no Phase 1 LLM dependency.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.