Idle-check sweep races with exclusion: revalidate IDLE_EXPIRED at claim time
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
GitHub issue: https://github.com/lablup/backend.ai/issues/13692
IdleCheckSweepHandler.execute() (sokovan/idle_check/handlers/sweep.py) receives the batch snapshotted by fetch_expired_idle_checks() and calls mark_sessions_for_termination() without re-reading the check rows. If a session idle-check exclusion (BA-7243, #13682 / #13328) flips a row from IDLE_EXPIRED to EXCLUDED between the snapshot and the termination request, the exclusion API reports success but the session is still terminated.
An exclusion landing before the snapshot does rescue the session (the sweep read filters on last_status == IDLE_EXPIRED); only the in-flight reconcile cycle (sub-second to seconds) loses the race, but the user-visible mismatch (success response, session terminated anyway) is real. Surfaced by the Copilot review on #13682.
Fix direction: revalidate at claim time in the sweep - before mark_sessions_for_termination, re-read the batch pairs' current last_status, or perform an atomic claim update conditioned on IDLE_EXPIRED, and terminate only sessions whose rows are still IDLE_EXPIRED. The exclusion side deliberately overwrites any phase; the fix belongs on the sweep side.
## Success Criteria
- [ ] exclusion flipping a row after the sweep snapshot but before termination: session is NOT terminated
- [ ] rows still IDLE_EXPIRED at claim time are terminated exactly as before
- [ ] regression test covering the snapshot-then-exclude interleaving
- [ ] pants test passes for affected packages
JIRA Issue: BA-7324
Contributor guide
Research direction
Start in sokovan/idle_check/handlers/sweep.py at IdleCheckSweepHandler.execute() and trace fetch_expired_idle_checks() through mark_sessions_for_termination(). Reproduce or test the snapshot-then-exclude interleaving, then run the affected packages with Pants. Done means rows still IDLE_EXPIRED are terminated as before, while rows changed to EXCLUDED before the claim are not terminated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100