lablup / lablup/backend.ai

Add restart backoff policy and max retry limit

Open
#9,809 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

Implement exponential backoff and maximum retry limit for inference session auto-restart to prevent infinite crash loops (similar to k8s CrashLoopBackOff).

Track restart attempts in session status_data:
- restart_count: number of restarts since last successful RUNNING period
- last_restart_at: timestamp of last restart
- Apply exponential backoff between restart attempts
- When max retry count exceeded, transition to TERMINATING
- Reset restart_count when session has been RUNNING healthily for a threshold period

Key files:
- manager/registry.py (restart_session - add backoff check)
- manager/data/session/types.py (status_data structure)
- manager/models/session/row.py (status_data column usage)

## Success Criteria
- [ ] First restart: immediate (no delay)
- [ ] Subsequent restarts: exponential backoff applied (e.g., 1s, 2s, 4s, 8s...)
- [ ] Max retry exceeded: session transitions to TERMINATING with reason indicating max retries
- [ ] status_data contains restart_count and last_restart_at after each restart
- [ ] restart_count resets after session runs healthily beyond threshold
- [ ] pants check passes for affected packages

JIRA Issue: BA-4957

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.