lablup / lablup/backend.ai

Idle-check sweep should revalidate IDLE_EXPIRED at claim time (race with exclusion)

Open
#13,692 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Problem

`IdleCheckSweepHandler.execute()` (`src/ai/backend/manager/sokovan/idle_check/handlers/sweep.py`) receives the batch snapshotted by `fetch_expired_idle_checks()` and calls `mark_sessions_for_termination()` without re-reading the check rows. If a session idle-check exclusion (BA-7243, #13682 / repository ops in #13328) flips a row from `IDLE_EXPIRED` to `EXCLUDED` between the snapshot and the termination request, the exclusion API reports success but the session is still terminated.

An exclusion that lands **before** the snapshot does rescue the session — the sweep read filters on `last_status == IDLE_EXPIRED`, so the overwritten row drops out. Only the in-flight reconcile cycle (sub-second to seconds) loses the race, but the user-visible mismatch (success response, session terminated anyway) is real.

Surfaced by the Copilot review on #13682: https://github.com/lablup/backend.ai/pull/13682

## Proposed fix

Revalidate at claim time in the sweep: before `mark_sessions_for_termination`, re-read the batch pairs' current `last_status` (or perform an atomic claim update conditioned on `IDLE_EXPIRED`) and terminate only sessions whose rows are still `IDLE_EXPIRED`.

## Scope notes

- The exclusion side deliberately overwrites any phase (a session already judged expired is "too late" to rescue by design); the fix belongs on the sweep side.
- Related: #13328 (BA-7120, repository ops), #13682 (BA-7243/BA-7244, API surface).

Contributor guide

Open the contributing guide

Research direction

Start in src/ai/backend/manager/sokovan/idle_check/handlers/sweep.py at IdleCheckSweepHandler.execute(), then trace fetch_expired_idle_checks() and mark_sessions_for_termination(). Reproduce or reason about the snapshot-to-termination race and ensure only rows still marked IDLE_EXPIRED are terminated; done means an exclusion that lands during the sweep prevents termination.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.