lablup / lablup/backend.ai

Auto-restart inference session containers on crash

Open
#9,806 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

When an inference (model serving) session's container dies (e.g., OOM), automatically restart the container within the same session instead of terminating the session.

Current behavior:
- Container OOM/crash → KernelTerminatedAnycastEvent → DETECT_KERNEL_TERMINATION PromotionSpec → Session RUNNING → TERMINATING (always)
- Route terminated → replica guarantee creates new Route + Session

Target behavior:
- Container OOM/crash → DETECT_KERNEL_TERMINATION detects INFERENCE session type → Session RUNNING → RESTARTING → restart_kernel() → RUNNING
- Route stays alive (DEGRADED during restart, HEALTHY after recovery)
- Backoff policy with max retry limit to prevent infinite crash loops

Existing infrastructure to leverage:
- RESTARTING state already defined in SessionStatus/KernelStatus
- restart_session() and restart_kernel() already implemented (manual API only)
- Intervention point: coordinator._process_promotion_scaling_group() or _handle_promotion_status_transitions()

JIRA Issue: BA-4954

Contributor guide

Open the contributing guide

Research direction

Start by tracing coordinator._process_promotion_scaling_group() and _handle_promotion_status_transitions(), then review the existing restart_session() and restart_kernel() implementations and the RESTARTING states in SessionStatus and KernelStatus. The change is done when an inference session detects a container crash, restarts within the same session with bounded backoff, and preserves the route through recovery.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.