Layr-Labs / Layr-Labs/d-inference
B2: sustained-load baseline for the three-Mac dev fleet
- Dominant language
- Go
- Stars
- 522
- Forks
- 113
- Avg merge
- 17h 26m
- Merged PRs (30d)
- 111
Description
Establish a load baseline for a three-Mac dev fleet, so later changes have something to be compared against.
Run the repo's load harness (`go test ./e2e/... -run TestBenchmark -v`) against the dev coordinator, plus concurrent streaming across all three providers.
### Observe and record
- TTFT and decode throughput, solo and under concurrency
- Continuous-batching scaling as batch size grows
- Queueing behaviour when all providers are busy
- 429 with `Retry-After` when the fleet is at capacity
- The `X-Timing` header decomposition (parse, reserve, route, queue, encrypt, dispatch, provider)
### Note on what this baseline is worth
Dev's object-storage CDN behaviour differs from prod's, so **model-download throughput numbers do not transfer**. Inference numbers are meaningful; download numbers are not. Record the distinction alongside the results.
### Acceptance criteria
- [ ] Baseline committed as a dated report under `docs/reports/`
- [ ] Queue and 429 paths observed, not just inferred
Contributor guide
Research direction
Start with the repo's load harness, `go test ./e2e/... -run TestBenchmark -v`, against the dev coordinator while streaming concurrently across all three providers. Record TTFT, decode throughput, batching, queueing, 429/`Retry-After`, and `X-Timing` results, noting that model-download throughput does not transfer from dev to prod. Commit the dated baseline report under `docs/reports/` with observed queue and 429 paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- documentation, performance
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100