Layr-Labs / Layr-Labs/d-inference
B3: recovery and failover drills against the dev fleet
- Dominant language
- Go
- Stars
- 522
- Forks
- 113
- Avg merge
- 17h 26m
- Merged PRs (30d)
- 111
Description
Exercise the failure paths deliberately, in the one environment where breaking them is free.
### Drills
1. **Provider killed mid-stream before first content** — should retry invisibly on another provider (`status:retry_precontent`), with no in-band SSE error reaching the client.
2. **Provider killed mid-stream after content has flowed** — should surface an in-band error; confirm the `inference.in_band_error` metric fires.
3. **Coordinator redeploy under load** — fleet reconnects; in-flight requests handled per the drain behaviour; confirm requests are not silently dropped.
4. **VM reboot** — persistent disk survives, and the state that lives only on disk (MicroMDM database, hard-untrust journal) is intact. Omitting the disk bind mount is what caused a past incident, so verify the mount explicitly.
5. **Repeated provider 5xx** — confirm the routing cooldown engages (two failures in 60s → 5-minute cooldown for that provider/model pair).
### Acceptance criteria
- [ ] All five drills run, with observed outcome recorded per drill
- [ ] Any divergence from documented behaviour filed as its own issue
Contributor guide
Research direction
Start by running the five listed recovery and failover drills against the dev fleet, checking the stated retry, in-band error, metric, reconnect, persistence, mount, and cooldown behaviors. Record the observed outcome for each drill; file any divergence from the documented behavior as a separate issue.
Written by the indexing model from the issue text.
Assessment
- Domain
- devops, infrastructure, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100