test(llm-router): measure sustained token-generation capacity and latency
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Description
Complete the deferred LLM request-router capacity and latency validation with
a real token-generating workload. The existing echo fixture emits a single
fixed completion token even when a larger token limit is requested, so its
end-to-end request duration cannot substantiate time to first token,
inter-token latency, sustained token throughput, or resource sizing.
Acceptance criteria
- Use a reproducible workload that generates enough tokens to sustain the
router data path. - Record workload identity, request shape, concurrency, duration, and token
accounting without retaining credentials or proprietary prompts. - Sample CPU and memory for both backend-router replicas and the concrete
Stargate replicas during steady state. - Report time to first token, inter-token latency, output-token throughput,
error rate, and end-to-end latency distributions. - Include a supported baseline comparison if a direct path remains available.
- Repeat after the documented single-router-pod failover and distinguish the
recovery interval from steady-state measurements. - Do not report echo-fixture timings as token-generation capacity.
This measurement is intentionally deferred until a suitable real workload is
available; it does not block the functional stable-stack validation.
By submitting this issue, you acknowledge that you are an assigned member of
the NVCF development team and agree to follow the project code of conduct and
contributing guidelines.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing echo fixture and the functional stable-stack validation, then trace the backend-router and Stargate replica entry points used by the test. Define a reproducible token-generating workload and collect the listed latency, throughput, error, CPU, memory, and failover measurements. Done means the report includes steady-state and post-failover results without echo timings or sensitive data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- ai, backend, performance, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100