feat(llm-routing): bound dynamic routing state with a canonical config cache
@along-2017 is already working on this.
Since Sep 3, 2026.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Description
Part of #536 (per-model routing configuration).
Bound the router's dynamic routing state. Canonicalize each valid expression
for identity (normalized algorithm name, parameters sorted by key) and share
one runtime definition per canonical configuration. Cap dynamic state per
router process: at most 1024 canonical definitions with LRU eviction and a
15 minute idle expiry, with the per-target load-balancer instances created
under those definitions counted against the same budget. In-flight requests
keep the state they started with; the first later request whose auth result
carries an update uses the new configuration.
This ticket makes expressions production-ready: stateful algorithms retain
their per-target state across requests because identical configurations reuse
one definition.
Definition of Done
- Identical expressions (regardless of parameter order or quoting style)
resolve to one shared cached definition - Cache respects the entry budget, LRU eviction, and idle expiry; target
cardinality cannot bypass the budget - Updating a model's expression requires no reload; in-flight requests are
isolated from the update - Tests cover canonical identity, bounds, eviction, no-reload updates, and
state isolation
Resources
- Parent: #536
- Blocked by: router expression parsing and overlay
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.