Layr-Labs / Layr-Labs/d-inference

B2: sustained-load baseline for the three-Mac dev fleet

Open
#1,071 0 comments 0 reactions 0 assignees View on GitHub
area:coordinator area:provider
Dominant language
Go
Stars
522
Forks
113
Avg merge
17h 26m
Merged PRs (30d)
111

Description

Establish a load baseline for a three-Mac dev fleet, so later changes have something to be compared against.

Run the repo's load harness (`go test ./e2e/... -run TestBenchmark -v`) against the dev coordinator, plus concurrent streaming across all three providers.

### Observe and record
- TTFT and decode throughput, solo and under concurrency
- Continuous-batching scaling as batch size grows
- Queueing behaviour when all providers are busy
- 429 with `Retry-After` when the fleet is at capacity
- The `X-Timing` header decomposition (parse, reserve, route, queue, encrypt, dispatch, provider)

### Note on what this baseline is worth
Dev's object-storage CDN behaviour differs from prod's, so **model-download throughput numbers do not transfer**. Inference numbers are meaningful; download numbers are not. Record the distinction alongside the results.

### Acceptance criteria
- [ ] Baseline committed as a dated report under `docs/reports/`
- [ ] Queue and 429 paths observed, not just inferred

Contributor guide

Open the contributing guide

Research direction

Start with the repo's load harness, `go test ./e2e/... -run TestBenchmark -v`, against the dev coordinator while streaming concurrently across all three providers. Record TTFT, decode throughput, batching, queueing, 429/`Retry-After`, and `X-Timing` results, noting that model-download throughput does not transfer from dev to prod. Commit the dated baseline report under `docs/reports/` with observed queue and 429 paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
documentation, performance
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.