NVIDIA / NVIDIA/nvcf

feat(llm-routing): bound dynamic routing state with a canonical config cache

Open
#1,404 0 comments 0 reactions 1 assignee View on GitHub

@along-2017 is already working on this.

Since Sep 3, 2026.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Description

Part of #536 (per-model routing configuration).

Bound the router's dynamic routing state. Canonicalize each valid expression
for identity (normalized algorithm name, parameters sorted by key) and share
one runtime definition per canonical configuration. Cap dynamic state per
router process: at most 1024 canonical definitions with LRU eviction and a
15 minute idle expiry, with the per-target load-balancer instances created
under those definitions counted against the same budget. In-flight requests
keep the state they started with; the first later request whose auth result
carries an update uses the new configuration.

This ticket makes expressions production-ready: stateful algorithms retain
their per-target state across requests because identical configurations reuse
one definition.

Definition of Done

  • Identical expressions (regardless of parameter order or quoting style)
    resolve to one shared cached definition
  • Cache respects the entry budget, LRU eviction, and idle expiry; target
    cardinality cannot bypass the budget
  • Updating a model's expression requires no reload; in-flight requests are
    isolated from the update
  • Tests cover canonical identity, bounds, eviction, no-reload updates, and
    state isolation

Resources

  • Parent: #536
  • Blocked by: router expression parsing and overlay

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.