lablup / lablup/backend.ai

Implement auto-restart trigger for inference sessions in DETECT_KERNEL_TERMINATION flow

Open
#9,808 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Modify the DETECT_KERNEL_TERMINATION promotion flow to conditionally restart inference sessions instead of terminating them.

Current flow in coordinator._process_promotion_scaling_group(): when any kernel is TERMINATED, the session is promoted to TERMINATING regardless of session type.

Target: check session_type == INFERENCE at the promotion point. If inference, route to RESTARTING state and trigger restart_session() instead of TERMINATING.

Key files:
- manager/sokovan/scheduler/factory.py (PromotionSpec for DETECT_KERNEL_TERMINATION)
- manager/sokovan/scheduler/coordinator.py (_process_promotion_scaling_group, _handle_promotion_status_transitions)
- manager/registry.py (restart_session)
- manager/event_dispatcher/handlers/session.py or kernel.py

## Success Criteria
- [ ] INFERENCE session with crashed container (exit code 137/OOM): session transitions RUNNING → RESTARTING → RUNNING (not TERMINATING)
- [ ] Non-INFERENCE session (INTERACTIVE/BATCH) with crashed container: session transitions RUNNING → TERMINATING (unchanged behavior)
- [ ] restart_kernel() is called on the agent for the crashed container
- [ ] Session status_history records the RESTARTING transition with timestamp
- [ ] pants check passes for affected packages

JIRA Issue: BA-4956

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.