Idle-check sweep should revalidate IDLE_EXPIRED at claim time (race with exclusion)
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Problem
`IdleCheckSweepHandler.execute()` (`src/ai/backend/manager/sokovan/idle_check/handlers/sweep.py`) receives the batch snapshotted by `fetch_expired_idle_checks()` and calls `mark_sessions_for_termination()` without re-reading the check rows. If a session idle-check exclusion (BA-7243, #13682 / repository ops in #13328) flips a row from `IDLE_EXPIRED` to `EXCLUDED` between the snapshot and the termination request, the exclusion API reports success but the session is still terminated.
An exclusion that lands **before** the snapshot does rescue the session — the sweep read filters on `last_status == IDLE_EXPIRED`, so the overwritten row drops out. Only the in-flight reconcile cycle (sub-second to seconds) loses the race, but the user-visible mismatch (success response, session terminated anyway) is real.
Surfaced by the Copilot review on #13682: https://github.com/lablup/backend.ai/pull/13682
## Proposed fix
Revalidate at claim time in the sweep: before `mark_sessions_for_termination`, re-read the batch pairs' current `last_status` (or perform an atomic claim update conditioned on `IDLE_EXPIRED`) and terminate only sessions whose rows are still `IDLE_EXPIRED`.
## Scope notes
- The exclusion side deliberately overwrites any phase (a session already judged expired is "too late" to rescue by design); the fix belongs on the sweep side.
- Related: #13328 (BA-7120, repository ops), #13682 (BA-7243/BA-7244, API surface).
Contributor guide
Research direction
Start in src/ai/backend/manager/sokovan/idle_check/handlers/sweep.py at IdleCheckSweepHandler.execute(), then trace fetch_expired_idle_checks() and mark_sessions_for_termination(). Reproduce or reason about the snapshot-to-termination race and ensure only rows still marked IDLE_EXPIRED are terminated; done means an exclusion that lands during the sweep prevents termination.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100