Layr-Labs / Layr-Labs/d-inference

B3: recovery and failover drills against the dev fleet

Open
#1,072 0 comments 0 reactions 0 assignees View on GitHub
area:coordinator
Dominant language
Go
Stars
522
Forks
113
Avg merge
17h 26m
Merged PRs (30d)
111

Description

Exercise the failure paths deliberately, in the one environment where breaking them is free.

### Drills
1. **Provider killed mid-stream before first content** — should retry invisibly on another provider (`status:retry_precontent`), with no in-band SSE error reaching the client.
2. **Provider killed mid-stream after content has flowed** — should surface an in-band error; confirm the `inference.in_band_error` metric fires.
3. **Coordinator redeploy under load** — fleet reconnects; in-flight requests handled per the drain behaviour; confirm requests are not silently dropped.
4. **VM reboot** — persistent disk survives, and the state that lives only on disk (MicroMDM database, hard-untrust journal) is intact. Omitting the disk bind mount is what caused a past incident, so verify the mount explicitly.
5. **Repeated provider 5xx** — confirm the routing cooldown engages (two failures in 60s → 5-minute cooldown for that provider/model pair).

### Acceptance criteria
- [ ] All five drills run, with observed outcome recorded per drill
- [ ] Any divergence from documented behaviour filed as its own issue

Contributor guide

Open the contributing guide

Research direction

Start by running the five listed recovery and failover drills against the dev fleet, checking the stated retry, in-band error, metric, reconnect, persistence, mount, and cooldown behaviors. Record the observed outcome for each drill; file any divergence from the documented behavior as a separate issue.

Written by the indexing model from the issue text.

Assessment

Domain
devops, infrastructure, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.