lablup / lablup/backend.ai

Idle-check sweep races with exclusion: revalidate IDLE_EXPIRED at claim time

Open
#13,693 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

GitHub issue: https://github.com/lablup/backend.ai/issues/13692

IdleCheckSweepHandler.execute() (sokovan/idle_check/handlers/sweep.py) receives the batch snapshotted by fetch_expired_idle_checks() and calls mark_sessions_for_termination() without re-reading the check rows. If a session idle-check exclusion (BA-7243, #13682 / #13328) flips a row from IDLE_EXPIRED to EXCLUDED between the snapshot and the termination request, the exclusion API reports success but the session is still terminated.

An exclusion landing before the snapshot does rescue the session (the sweep read filters on last_status == IDLE_EXPIRED); only the in-flight reconcile cycle (sub-second to seconds) loses the race, but the user-visible mismatch (success response, session terminated anyway) is real. Surfaced by the Copilot review on #13682.

Fix direction: revalidate at claim time in the sweep - before mark_sessions_for_termination, re-read the batch pairs' current last_status, or perform an atomic claim update conditioned on IDLE_EXPIRED, and terminate only sessions whose rows are still IDLE_EXPIRED. The exclusion side deliberately overwrites any phase; the fix belongs on the sweep side.

## Success Criteria
- [ ] exclusion flipping a row after the sweep snapshot but before termination: session is NOT terminated
- [ ] rows still IDLE_EXPIRED at claim time are terminated exactly as before
- [ ] regression test covering the snapshot-then-exclude interleaving
- [ ] pants test passes for affected packages

JIRA Issue: BA-7324

Contributor guide

Open the contributing guide

Research direction

Start in sokovan/idle_check/handlers/sweep.py at IdleCheckSweepHandler.execute() and trace fetch_expired_idle_checks() through mark_sessions_for_termination(). Reproduce or test the snapshot-then-exclude interleaving, then run the affected packages with Pants. Done means rows still IDLE_EXPIRED are terminated as before, while rows changed to EXCLUDED before the claim are not terminated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.