Discount cached prefill cost in WaitAndWiden routing
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Problem
WaitAndWiden estimates the full input-prefill cost for a backend selected by cache-affinity hashing. For a continuing session, most input tokens may already be in that backend cache. Treating all of them as cold can place the affinity backend in a later TTFT bucket and route the request to a cold backend too early.
Proposed behavior
Allow WaitAndWiden to scale the current request prefill estimate for cache-affinity candidates. Keep queued work unchanged, keep cold candidates at the full input cost, and retain the current behavior by default.
Acceptance criteria
- The scale is bounded to the inclusive range 0 through 1.
- Only affinity candidates receive the discounted current-request prefill cost.
- Existing configurations behave unchanged when the setting is omitted.
- A full affinity candidate can anchor the first TTFT bucket, while cold candidates become eligible through normal widening.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the WaitAndWiden routing entry point and trace cache-affinity candidate selection and prefill-cost estimation. Confirm the configuration path and focused routing tests, then verify the scale bounds, affinity-only discount, unchanged defaults, and TTFT bucket behavior described in the acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100