hoangsonww / hoangsonww/Forge-Agentic-Coding-CLI
Feature: Model Routing Calibration Suite with Local Benchmarks and Role Fit Reports
- Dominant language
- TypeScript
- Stars
- 23
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Add a calibration workflow that benchmarks available local and hosted model providers, measures role-specific behavior, and produces actionable routing recommendations for planner, architect, executor, reviewer, debugger, memory, and fast roles.
## Problem / Opportunity
Forge already has provider discovery, local model auto-adaptation, circuit breakers, rate limits, prompt caching, cost tracking, and a local model catalogue. The current routing path is mostly configuration-driven plus availability/fallback behavior. Users running Ollama, LM Studio, vLLM, llama.cpp, Anthropic, or OpenAI-compatible endpoints need a project-aware way to answer:
- Which installed model is best for executor work in this repository?
- Which provider is fastest for planner or reviewer tasks?
- What context window and latency tradeoffs should I expect?
- How should `models.fast`, `models.code`, `models.planner`, and `models.balanced` be configured?
- Did a provider or model get worse after an upgrade?
A calibration suite would make local-first routing tangible and reduce trial-and-error setup.
## Proposed Feature
Introduce `forge model calibrate` and matching dashboard support that:
- Detects available providers and installed models.
- Runs a bounded benchmark suite using small, deterministic tasks modeled after Forge roles.
- Measures latency, streaming behavior where available, success/failure, context handling, token usage, cost, and tool-call JSON reliability.
- Produces a role-fit report and optional config patch proposal.
- Stores historical calibration runs so users can compare before/after results.
- Integrates with `forge doctor` to surface stale or missing calibration data.
## Scope
Expected implementation areas:
- `src/models/router.ts`, `src/models/adapter.ts`, and `src/models/local-catalog.ts` for route recommendation output.
- `src/models/cost.ts`, `src/models/rate-limit.ts`, and `src/models/circuit-breaker.ts` for measurement and failure context.
- New CLI command under `src/cli/commands/model.ts`.
- Persistence support in `src/persistence/index-db.ts` for calibration results.
- Dashboard additions in `src/ui/server.ts` and `src/ui/public/`.
- Docs in `docs/SETUP.md`, `docs/CLI-REFERENCE.md`, and `docs/ARCHITECTURE.md`.
## Acceptance Criteria
- [ ] `forge model calibrate` runs without network calls unless a hosted provider is explicitly configured and available.
- [ ] Calibration can target all providers or a single provider/model.
- [ ] Results include per-role latency, failure rate, context estimate, JSON/tool-call reliability, cost where available, and recommendation rationale.
- [ ] The workflow proposes a config update without applying it unless the user confirms.
- [ ] Historical calibration reports can be listed and compared.
- [ ] `forge doctor` warns when configured models are unavailable or calibration is stale.
- [ ] Tests mock provider responses and do not introduce live network calls.
- [ ] Documentation explains how to interpret role-fit scores and when to prefer local vs hosted providers.
## Non-Goals
- Creating a public leaderboard.
- Uploading prompts, outputs, or benchmark data to a remote service.
- Replacing manual model configuration for advanced users.
- Performing large synthetic evals that take minutes by default.
## Dependencies / Risks
- Benchmark prompts must be small enough to preserve Forge's fast local workflow.
- Hosted model tests must respect user cost expectations and require clear confirmation.
- The scoring model should be transparent and stable, not a black-box ranking.
- Calibration should not pollute normal prompt cache or cost accounting unless explicitly marked as calibration.
## Open Questions
- Should calibration include project-specific samples from the current repository, or only built-in synthetic tasks?
- Should the first version support streaming metrics?
- What default freshness window should `forge doctor` use for calibration results?
- Should recommendations include mode-specific routing, such as different choices for `offline-safe` and `heavy`?
Contributor guide
Assessment
This issue has not been assessed yet.