Auto-restart inference session containers on crash
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
When an inference (model serving) session's container dies (e.g., OOM), automatically restart the container within the same session instead of terminating the session.
Current behavior:
- Container OOM/crash → KernelTerminatedAnycastEvent → DETECT_KERNEL_TERMINATION PromotionSpec → Session RUNNING → TERMINATING (always)
- Route terminated → replica guarantee creates new Route + Session
Target behavior:
- Container OOM/crash → DETECT_KERNEL_TERMINATION detects INFERENCE session type → Session RUNNING → RESTARTING → restart_kernel() → RUNNING
- Route stays alive (DEGRADED during restart, HEALTHY after recovery)
- Backoff policy with max retry limit to prevent infinite crash loops
Existing infrastructure to leverage:
- RESTARTING state already defined in SessionStatus/KernelStatus
- restart_session() and restart_kernel() already implemented (manual API only)
- Intervention point: coordinator._process_promotion_scaling_group() or _handle_promotion_status_transitions()
JIRA Issue: BA-4954
Contributor guide
Research direction
Start by tracing coordinator._process_promotion_scaling_group() and _handle_promotion_status_transitions(), then review the existing restart_session() and restart_kernel() implementations and the RESTARTING states in SessionStatus and KernelStatus. The change is done when an inference session detects a container crash, restarts within the same session with bounded backoff, and preserves the route through recovery.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100