NVIDIA / NVIDIA/nvcf

feat(llm-routing): parse routing expressions and apply tuning overlays in the router

Open
#1,403 0 comments 0 reactions 1 assignee View on GitHub

@along-2017 is already working on this.

Since Sep 3, 2026.

Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

Description

Part of #536 (per-model routing configuration).

Teach the request router to accept a routing method expression, an algorithm
name plus scalar tuning parameters, in the routing-method header. Parse it as a
profile of RFC 8941 Item with Parameters. Resolve the algorithm through the
existing configured/default/request-algorithm chain, clone the resolved base
configuration, and overlay the supplied parameters; omitted fields keep their
static values. Reject a semantically invalid expression (unknown method or
parameter, incompatible parameter, invalid value, inert combination) with HTTP
400 before provider selection, using the existing rejection logging shape.

Method-only values keep today's behavior. This ticket builds definitions per
request without identity reuse; production enablement waits for the cache
ticket.

Definition of Done

  • Well-formed expressions for every supported algorithm parse and route
    with the requested overlays applied; omitted fields keep static values
  • Method-only values behave exactly as today
  • Semantically invalid expressions return 400 before provider selection
    with a logged reason
  • Every direct scalar load-balancer configuration field is supported and
    its algorithm applicability enumerated
  • Table-driven tests cover grammar acceptance, rejection, and overlay
    semantics

Resources

  • Parent: #536
  • Blocked by: none

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.