CopilotKit / CopilotKit/outpost

Pathfinder: synthetic snippet scores make the heuristic-confidence fallback near-constant HIGH

Open
#122 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: integrations enhancement priority: low roadmap roadmap: later
Dominant language
TypeScript
Stars
7
Forks
3
Avg merge
7d 16h
Merged PRs (30d)
15

Description

Spun out of the #112 review (non-blocking).

parseSnippets synthesizes score: Math.max(0.5, 1 - i * 0.05), so results[0].score is always 1.0. ConfidenceScorer.heuristicScore derives from topScore * 0.4 + avgScore * 0.4 + min(count*0.03, 0.2); with the default 8 server-filtered results this lands ~0.93 → HIGH essentially whenever any results return, regardless of real match quality.

Impact is bounded — the primary Claude scorer (reads generated text + snippet content) is unaffected; this only drives the degraded/fallback heuristic path and any pre-filter. But it's a behavior change from the old JSON path, where genuine similarity (0.3–0.92) produced graduated heuristics. The tradeoff was made consciously (results are already server-filtered by min_score), so this is a "revisit if it matters" item, not a bug.

Options: (a) if copilotkit-docs-mcp exposes a numeric relevance anywhere in the payload, thread it through instead of the synthetic rank; (b) otherwise, damp the heuristic's reliance on topScore when scores are synthetic, or flag synthetic-scored results so heuristicScore can treat them differently.

Files: packages/outpost/ai/src/pathfinder.ts (parseSnippets), packages/outpost/ai/src/confidence.ts (heuristicScore).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/outpost/ai/src/pathfinder.ts, especially parseSnippets, and packages/outpost/ai/src/confidence.ts, especially heuristicScore. Trace whether copilotkit-docs-mcp exposes numeric relevance in its payload and compare the synthetic scores with the old JSON path. Done means the fallback or pre-filter no longer produces near-constant HIGH confidence when match quality differs.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.