NVIDIA / NVIDIA/nvcf

test(llm-router): measure sustained token-generation capacity and latency

Open
#1,230 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Description

Complete the deferred LLM request-router capacity and latency validation with
a real token-generating workload. The existing echo fixture emits a single
fixed completion token even when a larger token limit is requested, so its
end-to-end request duration cannot substantiate time to first token,
inter-token latency, sustained token throughput, or resource sizing.

Acceptance criteria

  • Use a reproducible workload that generates enough tokens to sustain the
    router data path.
  • Record workload identity, request shape, concurrency, duration, and token
    accounting without retaining credentials or proprietary prompts.
  • Sample CPU and memory for both backend-router replicas and the concrete
    Stargate replicas during steady state.
  • Report time to first token, inter-token latency, output-token throughput,
    error rate, and end-to-end latency distributions.
  • Include a supported baseline comparison if a direct path remains available.
  • Repeat after the documented single-router-pod failover and distinguish the
    recovery interval from steady-state measurements.
  • Do not report echo-fixture timings as token-generation capacity.

This measurement is intentionally deferred until a suitable real workload is
available; it does not block the functional stable-stack validation.


By submitting this issue, you acknowledge that you are an assigned member of
the NVCF development team and agree to follow the project code of conduct and
contributing guidelines.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing echo fixture and the functional stable-stack validation, then trace the backend-router and Stargate replica entry points used by the test. Define a reproducible token-generating workload and collect the listed latency, throughput, error, CPU, memory, and failover measurements. Done means the report includes steady-state and post-failover results without echo timings or sensitive data.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
ai, backend, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.