OpenBMB / OpenBMB/PilotDeck

bug(config): validation misses router.stats.baselineModel

Open
#249 0 comments 0 reactions 1 assignee View on GitHub

@Gucc111 is already working on this.

Since Jul 10, 2026.

Dominant language
TypeScript
Stars
4k
Forks
453
Avg merge
12h 30m
Merged PRs (30d)
46

Description

UI-related note: this touches the Settings/UI path and may be lower priority during the ongoing frontend refactor.

Summary

UI-side config validation does not check router.stats.baselineModel, even though the core router parser treats it as a strict provider/model ref.

Why it matters

If a provider is deleted or renamed, router.stats.baselineModel can still point to the old provider. Settings validation can pass and save the config, but core load fails with a fatal router diagnostic. This is another save-success/load-failure config path.

Evidence

  • src/router/config/schema.ts:61 defines baselineModel as the provider/model ref used for saved-cost baseline calculation.
  • src/router/config/parseRouterConfig.ts:76 parses router stats.
  • src/router/config/parseRouterConfig.ts:491 enters parseStats.
  • src/router/config/parseRouterConfig.ts:532 validates raw.baselineModel through optionalRef.
  • src/router/config/parseRouterConfig.ts:566 turns ref resolution issues into fatal diagnostics.
  • src/router/config/schema.ts:229 looks up the referenced provider.
  • src/router/config/schema.ts:232 emits ROUTER_REF_PROVIDER_NOT_FOUND for an unknown provider.
  • ui/server/services/pilotdeckConfig.js:191 starts validateRouterModelRefs, but only checks scenarios, fallback, and tokenSaver refs.
  • ui/server/services/pilotdeckConfig.js:248 calls that incomplete validator.
  • ui/server/services/pilotdeckConfig.js:511 starts provider deletion/orphan cleanup.
  • ui/server/services/pilotdeckConfig.js:525 repairs scenario refs.
  • ui/server/services/pilotdeckConfig.js:530 repairs fallback refs.
  • ui/server/services/pilotdeckConfig.js:535 repairs tokenSaver refs.
  • ui/server/services/pilotdeckConfig.js:548 returns without repairing router.stats.baselineModel.
  • ui/src/components/settings/view/tabs/PilotDeckConfigTab.tsx:176 models router.stats in the Settings config type.
  • ui/src/components/settings/view/tabs/PilotDeckConfigTab.tsx:2202 edits router.stats.modelPricing.
  • ui/src/components/settings/view/tabs/PilotDeckConfigTab.tsx:3021 toggles router.stats.enabled.
  • ui/server/routes/config.js:120 and ui/server/routes/config.js:124 route UI saves through the config writer.

Validation

Validation level: dynamic reproduction plus source-control-flow confirmation.

Minimal reproduction:

  1. Create a config with a valid agent.model.
  2. Set router.stats.baselineModel: old/m.
  3. Ensure model.providers.old does not exist.
  4. Run UI validation and then core router parsing on the same config.

Key output:

  • validatePilotDeckConfig(...) returns valid: true.
  • parseRouterConfig(...) emits fatal diagnostic:
    • router.stats.baselineModel references unknown provider old.

Boundary: this validates the validator/parser mismatch directly. It does not require the stats collector to write data. It does not depend on the frontend rendering path beyond the save route using the same validation service.

Expected behavior

Every model ref accepted by Settings should be validated against configured providers before save. router.stats.baselineModel should also participate in provider rename and provider deletion cleanup, matching the strict parser behavior.

Existing coverage checked

No matching fix was found.

Checked adjacent work:

  • PR #185 validates model provider shape, but does not cover router.stats.baselineModel.
  • PR #187 fixes object router refs, but does not cover this field.

Search terms checked included router.stats.baselineModel, validateRouterModelRefs stats, and baselineModel references unknown provider.

Suggested fix

Add router.stats.baselineModel to:

  • UI/server save-time router ref validation.
  • Provider rename rewrite logic.
  • Provider deletion/orphan cleanup logic.

Prefer using the same model-ref validation and rewrite helpers already used for router.scenarios, router.fallback, and router.tokenSaver.

Suggested tests

  • UI/server validation rejects router.stats.baselineModel: old/m when old is not configured.
  • Provider rename rewrites router.stats.baselineModel.
  • Provider deletion/orphan cleanup repairs or removes stale router.stats.baselineModel.
  • A config accepted by UI validation should not produce fatal ROUTER_REF_PROVIDER_NOT_FOUND diagnostics for router.stats.baselineModel in the core parser.

Submitted with Codex.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.