bug(router): explicit stats.baselineModel is overridden by the default scenario
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 4k
- Forks
- 453
- Avg merge
- 12h 30m
- Merged PRs (30d)
- 46
Description
Summary
router.stats.baselineModel is parsed, but createRouterRuntime overrides it with router.scenarios.default whenever a default scenario exists.
Why it matters
The purpose of router.stats.baselineModel is to define the no-router baseline used for saved-cost and baseline-cost calculations. Most router configs have a default scenario, so the explicit baseline is silently ignored in normal use. The dashboard or stats output can then report savings against the routed default model instead of the user-selected baseline model.
Evidence
src/router/config/schema.ts:61documentsbaselineModelas the provider/model ref used for saved-cost baseline calculation.src/router/config/parseRouterConfig.ts:491entersparseStats.src/router/config/parseRouterConfig.ts:532parsesraw.baselineModel.src/router/config/parseRouterConfig.ts:538returns the parsedbaselineModel.src/router/RouterRuntime.ts:90constructsTokenStatsCollector.src/router/RouterRuntime.ts:93prefersconfig.scenarios.defaultas the collector baseline whenever a default scenario exists.src/router/RouterRuntime.ts:95only falls back toconfig.stats?.baselineModelwhen there is no default scenario.src/router/stats/TokenStatsCollector.ts:68stores the collector baseline fromconfig?.baselineModel.src/router/stats/TokenStatsCollector.ts:317uses that baseline during baseline-cost calculation.src/router/stats/TokenStatsCollector.ts:321selects the baseline provider.src/router/stats/TokenStatsCollector.ts:322selects the baseline model.
Validation
Validation level: dynamic reproduction.
Minimal reproduction:
- Create router config with:
router.scenarios.default = cheap/crouter.stats.baselineModel = exp/e
- Call
createRouterRuntime(...). - Inspect the created stats collector baseline.
Key output:
- Parsed config preserves the explicit baseline.
- Runtime stats collector receives
{ provider: "cheap", model: "c" }. - Expected explicit baseline was
{ provider: "exp", model: "e" }.
Boundary: this repro does not need to perform a real model call. It validates the runtime construction path where stats collector config is derived. It does not change scenario routing behavior; it only observes which baseline is passed to stats.
Expected behavior
An explicitly configured router.stats.baselineModel should take precedence over router.scenarios.default. The default scenario should only be used as a fallback baseline when no explicit stats baseline is configured.
Existing coverage checked
No matching fix was found.
Checked adjacent work:
- PR #208 modifies
RouterRuntimefor resilience, retry, and provider health tracking, but does not change stats baseline precedence and does not add tests for explicitstats.baselineModel.
Search terms checked included router.stats.baselineModel, TokenStatsCollector baselineModel, and savedCost baseline.
Suggested fix
Change the stats collector construction priority in createRouterRuntime so it uses:
config.stats?.baselineModel, when explicitly present.config.scenarios?.default, only as fallback.- No baseline override otherwise.
This keeps routing behavior unchanged while making the stats contract match the parser/schema contract.
Suggested tests
- Explicit
stats.baselineModelwins whenscenarios.defaultalso exists. scenarios.defaultis used as fallback whenstats.baselineModelis absent.- Explicit
stats.baselineModelworks when no default scenario exists. - Baseline-cost calculation uses the explicit baseline provider/model after runtime construction.
Submitted with Codex.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at createRouterRuntime in src/router/RouterRuntime.ts and trace the baseline passed to TokenStatsCollector. Read the parsing path in src/router/config/parseRouterConfig.ts and baseline usage in src/router/stats/TokenStatsCollector.ts, then add focused coverage for explicit and fallback baselines. Done means runtime construction preserves an explicit stats baseline and the fallback cases remain covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- analytics, backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100