Implement auto-restart trigger for inference sessions in DETECT_KERNEL_TERMINATION flow
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
Modify the DETECT_KERNEL_TERMINATION promotion flow to conditionally restart inference sessions instead of terminating them.
Current flow in coordinator._process_promotion_scaling_group(): when any kernel is TERMINATED, the session is promoted to TERMINATING regardless of session type.
Target: check session_type == INFERENCE at the promotion point. If inference, route to RESTARTING state and trigger restart_session() instead of TERMINATING.
Key files:
- manager/sokovan/scheduler/factory.py (PromotionSpec for DETECT_KERNEL_TERMINATION)
- manager/sokovan/scheduler/coordinator.py (_process_promotion_scaling_group, _handle_promotion_status_transitions)
- manager/registry.py (restart_session)
- manager/event_dispatcher/handlers/session.py or kernel.py
## Success Criteria
- [ ] INFERENCE session with crashed container (exit code 137/OOM): session transitions RUNNING → RESTARTING → RUNNING (not TERMINATING)
- [ ] Non-INFERENCE session (INTERACTIVE/BATCH) with crashed container: session transitions RUNNING → TERMINATING (unchanged behavior)
- [ ] restart_kernel() is called on the agent for the crashed container
- [ ] Session status_history records the RESTARTING transition with timestamp
- [ ] pants check passes for affected packages
JIRA Issue: BA-4956
Contributor guide
Assessment
This issue has not been assessed yet.