OpenBMB / OpenBMB/PilotDeck

bug(router): explicit stats.baselineModel is overridden by the default scenario

Open Beginner friendly
#250 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
4k
Forks
453
Avg merge
12h 30m
Merged PRs (30d)
46

Description

Summary

router.stats.baselineModel is parsed, but createRouterRuntime overrides it with router.scenarios.default whenever a default scenario exists.

Why it matters

The purpose of router.stats.baselineModel is to define the no-router baseline used for saved-cost and baseline-cost calculations. Most router configs have a default scenario, so the explicit baseline is silently ignored in normal use. The dashboard or stats output can then report savings against the routed default model instead of the user-selected baseline model.

Evidence

  • src/router/config/schema.ts:61 documents baselineModel as the provider/model ref used for saved-cost baseline calculation.
  • src/router/config/parseRouterConfig.ts:491 enters parseStats.
  • src/router/config/parseRouterConfig.ts:532 parses raw.baselineModel.
  • src/router/config/parseRouterConfig.ts:538 returns the parsed baselineModel.
  • src/router/RouterRuntime.ts:90 constructs TokenStatsCollector.
  • src/router/RouterRuntime.ts:93 prefers config.scenarios.default as the collector baseline whenever a default scenario exists.
  • src/router/RouterRuntime.ts:95 only falls back to config.stats?.baselineModel when there is no default scenario.
  • src/router/stats/TokenStatsCollector.ts:68 stores the collector baseline from config?.baselineModel.
  • src/router/stats/TokenStatsCollector.ts:317 uses that baseline during baseline-cost calculation.
  • src/router/stats/TokenStatsCollector.ts:321 selects the baseline provider.
  • src/router/stats/TokenStatsCollector.ts:322 selects the baseline model.

Validation

Validation level: dynamic reproduction.

Minimal reproduction:

  1. Create router config with:
    • router.scenarios.default = cheap/c
    • router.stats.baselineModel = exp/e
  2. Call createRouterRuntime(...).
  3. Inspect the created stats collector baseline.

Key output:

  • Parsed config preserves the explicit baseline.
  • Runtime stats collector receives { provider: "cheap", model: "c" }.
  • Expected explicit baseline was { provider: "exp", model: "e" }.

Boundary: this repro does not need to perform a real model call. It validates the runtime construction path where stats collector config is derived. It does not change scenario routing behavior; it only observes which baseline is passed to stats.

Expected behavior

An explicitly configured router.stats.baselineModel should take precedence over router.scenarios.default. The default scenario should only be used as a fallback baseline when no explicit stats baseline is configured.

Existing coverage checked

No matching fix was found.

Checked adjacent work:

  • PR #208 modifies RouterRuntime for resilience, retry, and provider health tracking, but does not change stats baseline precedence and does not add tests for explicit stats.baselineModel.

Search terms checked included router.stats.baselineModel, TokenStatsCollector baselineModel, and savedCost baseline.

Suggested fix

Change the stats collector construction priority in createRouterRuntime so it uses:

  1. config.stats?.baselineModel, when explicitly present.
  2. config.scenarios?.default, only as fallback.
  3. No baseline override otherwise.

This keeps routing behavior unchanged while making the stats contract match the parser/schema contract.

Suggested tests

  • Explicit stats.baselineModel wins when scenarios.default also exists.
  • scenarios.default is used as fallback when stats.baselineModel is absent.
  • Explicit stats.baselineModel works when no default scenario exists.
  • Baseline-cost calculation uses the explicit baseline provider/model after runtime construction.

Submitted with Codex.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at createRouterRuntime in src/router/RouterRuntime.ts and trace the baseline passed to TokenStatsCollector. Read the parsing path in src/router/config/parseRouterConfig.ts and baseline usage in src/router/stats/TokenStatsCollector.ts, then add focused coverage for explicit and fallback baselines. Done means runtime construction preserves an explicit stats baseline and the fallback cases remain covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
analytics, backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.