lablup / lablup/backend.ai

Surface multi-node scheduling context in scheduling failure messages

Open
#11,746 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

## Problem

When a multi-node session fails during agent selection, the recorded failure message only reflects the \***last**\* kernel iteration that could not be placed. Operators cannot tell from `status_data` / `scheduling_history`:

- whether it was a multi-node attempt (cluster_mode, cluster_size)
- which kernel index in the batch failed
- which agents had already received earlier kernels in the same session (in-batch successful selections)
- whether each agent's `available=N` reflects the original agent state or the post-diff state after this session's earlier kernels consumed capacity

Example today:

```
Unavailable : No agents can be allocated at this time. (
no available agents. Details:
1x Agent i-brachio01 has insufficient resources: cuda.shares: requested=1, available=0;
1x Agent i-brachio02 has insufficient resources: cuda.shares: requested=1, available=0
)
```

Indistinguishable cases:
- agents truly saturated regardless of this session → cluster admin must add capacity
- earlier kernels of **this** session ate the capacity → user could try smaller cluster_size

## Root Cause Trace

1. `selectors/selector.py:355-374` — `select_agents_for_batch_requirements` loops `for resource_req in resource_requirements` and discards `selections: list[AgentSelection]` (prior successes) when `_select_agent_tracker_for_requirements` raises
2. `selectors/selector.py:425-428` — flattens per-agent errors into a single `; `-joined string via `NoAvailableAgentError`; no iteration index, no cluster context
3. `selectors/exceptions.py:84-109` — `InsufficientResourcesError` carries `requested_slots`, `available_slots`, `occupied_slots` but `__str__` only outputs `requested=X, available=Y`; agent-occupied vs in-batch-diff portion of `occupied_slots` not exposed
4. `provisioner.py:256-272` — final flattening: `SchedulingFailure.msg = str(e)`, which is what reaches `status_data`

The enrichment data exists internally (`AgentStateTracker.additional_slots`, `criteria.session_metadata.cluster_mode`, `criteria.kernel_requirements`, the in-progress `selections` list) but never crosses the exception boundary.

## Approach

1. \***`selectors/exceptions.py` — `InsufficientResourcesError.__str__`**\*: include both the agent's intrinsic occupied state and the in-batch diff portion contributed by this session's prior kernels, so `available=0` is decomposable.
2. \***`selectors/selector.py:355-428` — `select_agents_for_batch_requirements`**\*: capture `iteration_index`, `total_iterations` (cluster_size), and `prior_selections: list[(kernel_id, agent_id)]` before delegating. On failure, wrap the inner `NoAvailableAgentError` with this multi-node context (either via a new exception subclass `MultiNodeAllocationFailure` or by re-raising with an enriched message).
3. \***`provisioner.py:256-272`**\*: when materializing `SchedulingFailure`, ensure the enriched context (cluster_mode, cluster_size, prior selections) is preserved in `msg`. Consider whether structured fields on `SchedulingFailure` (beyond `msg: str`) are warranted so WebUI can render them — see Epic BA-6147 follow-up.

## Out of Scope

- Changing the `SchedulingFailure` schema beyond what's needed for the message (structured fields can come later)
- Validator-side error messages (this story focuses on agent selection failures)

## Success Criteria

### Multi-node context in message
- [ ] Multi-node session (cluster_size > 1) fails agent selection: error message contains `cluster_mode` and `cluster_size`
- [ ] Error message identifies which kernel index in the batch failed (e.g., `kernel 3/4`)
- [ ] Error message lists prior successful selections in the same batch (e.g., `kernel 1 → i-brachio01, kernel 2 → i-brachio02`)

### Per-agent state decomposition
- [ ] `InsufficientResourcesError` message includes both agent-intrinsic occupied and in-batch diff (this session's prior consumption), so operators can distinguish saturated agent vs in-batch consumption

### Persistence
- [ ] The enriched message is preserved in `SchedulingFailure.msg` and reaches `status_data` / `scheduling_history` rows for the failing session

### Regression
- [ ] Single-node sessions (cluster_size=1) message format unchanged — no multi-node context appended
- [ ] No new mandatory fields on `SchedulingFailure` that break existing consumers
- [ ] pants test passes for affected packages

JIRA Issue: BA-6148

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.