CopilotKit / CopilotKit/outpost
Pathfinder: synthetic snippet scores make the heuristic-confidence fallback near-constant HIGH
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 3
- Avg merge
- 7d 16h
- Merged PRs (30d)
- 15
Description
Spun out of the #112 review (non-blocking).
parseSnippets synthesizes score: Math.max(0.5, 1 - i * 0.05), so results[0].score is always 1.0. ConfidenceScorer.heuristicScore derives from topScore * 0.4 + avgScore * 0.4 + min(count*0.03, 0.2); with the default 8 server-filtered results this lands ~0.93 → HIGH essentially whenever any results return, regardless of real match quality.
Impact is bounded — the primary Claude scorer (reads generated text + snippet content) is unaffected; this only drives the degraded/fallback heuristic path and any pre-filter. But it's a behavior change from the old JSON path, where genuine similarity (0.3–0.92) produced graduated heuristics. The tradeoff was made consciously (results are already server-filtered by min_score), so this is a "revisit if it matters" item, not a bug.
Options: (a) if copilotkit-docs-mcp exposes a numeric relevance anywhere in the payload, thread it through instead of the synthetic rank; (b) otherwise, damp the heuristic's reliance on topScore when scores are synthetic, or flag synthetic-scored results so heuristicScore can treat them differently.
Files: packages/outpost/ai/src/pathfinder.ts (parseSnippets), packages/outpost/ai/src/confidence.ts (heuristicScore).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with packages/outpost/ai/src/pathfinder.ts, especially parseSnippets, and packages/outpost/ai/src/confidence.ts, especially heuristicScore. Trace whether copilotkit-docs-mcp exposes numeric relevance in its payload and compare the synthetic scores with the old JSON path. Done means the fallback or pre-filter no longer produces near-constant HIGH confidence when match quality differs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100