Add restart backoff policy and max retry limit
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
Implement exponential backoff and maximum retry limit for inference session auto-restart to prevent infinite crash loops (similar to k8s CrashLoopBackOff).
Track restart attempts in session status_data:
- restart_count: number of restarts since last successful RUNNING period
- last_restart_at: timestamp of last restart
- Apply exponential backoff between restart attempts
- When max retry count exceeded, transition to TERMINATING
- Reset restart_count when session has been RUNNING healthily for a threshold period
Key files:
- manager/registry.py (restart_session - add backoff check)
- manager/data/session/types.py (status_data structure)
- manager/models/session/row.py (status_data column usage)
## Success Criteria
- [ ] First restart: immediate (no delay)
- [ ] Subsequent restarts: exponential backoff applied (e.g., 1s, 2s, 4s, 8s...)
- [ ] Max retry exceeded: session transitions to TERMINATING with reason indicating max retries
- [ ] status_data contains restart_count and last_restart_at after each restart
- [ ] restart_count resets after session runs healthily beyond threshold
- [ ] pants check passes for affected packages
JIRA Issue: BA-4957
Contributor guide
Assessment
This issue has not been assessed yet.