lablup / lablup/backend.ai

Improve scheduling history visibility in manager

Open
#11,745 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Background

Scheduling failures are recorded into `status_data` and the `scheduling_history` table as a flattened string (`SchedulingFailure.msg = str(e)`). WebUI / CLI render this string verbatim. Today the operator experience has two concrete pain points:

1. \***Multi-node context is lost.**\* When a multi-node session fails mid-batch (e.g., 3rd of 4 kernels can't fit on any agent), the surfaced error reflects only the last failing iteration. Cluster size, the kernel index that failed, and which agents had already received earlier kernels in the same session are discarded at the exception boundary. Operators can't tell whether `available=0` means the agent is genuinely full or earlier kernels in the same session consumed the slot.
2. \***Single-line `;`-joined messages are unreadable.**\* Per-slot details (`InsufficientResourcesError.__str__`) and per-agent aggregations (`_select_agent_tracker_for_requirements`) both use `; ` as separator, so long failure messages run into a single dense line. WebUI/CLI cannot present these structurally.

## Scope

Two child Stories:
- Surface multi-node scheduling context in failure messages
- Switch failure-message separators from `;` to `\n` so WebUI/CLI can split per line

Both target the same files (`selectors/exceptions.py`, `selectors/selector.py`, `provisioner.py`) but in non-overlapping ways. Recommended landing order: separator change first, then the multi-node enrichment uses newline from the start.

WebUI / CLI side will adapt to the newline-delimited format once the manager change ships.

## Out of Scope

- WebUI / CLI rendering changes (tracked separately on those repos)
- Functional changes to scheduling decisions

JIRA Issue: BA-6147

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.