CopilotKit / CopilotKit/outpost

Synthesized positional scores inflate retrieval confidence once code search returns hits

Open
#243 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: ai roadmap roadmap: next
Dominant language
TypeScript
Stars
7
Forks
3
Avg merge
7d 16h
Merged PRs (30d)
15

Description

Found by the review of PR #242 (Phase 2, code search). Not fixed there because it changes confidence behaviour on every ticket and wants measuring rather than guessing.

parseSnippets synthesizes a positional score for text-format results — score: Math.max(0.5, 1 - i * 0.05) (pathfinder.ts) — because the MCP text format carries no numeric relevance. That was tolerable while only search-docs produced them. Now search-code produces its own independent list, so the top code hit gets 1.0 regardless of how marginal it is, and min_score defaults to 0.3.

Two consumers read those numbers:

  • confidence.heuristicScoretopScore * 0.4 + avgScore * 0.4 + min(count * 0.03, 0.2)
  • generator.assessConfidenceavg + min(count * 0.05, 0.15)

So retrieval confidence is inflated for any question that returns any code hit, including irrelevant ones. That is the opposite of what the groundedness work in this module was for.

PR #242 caps the merged list at defaultLimit, which fixes the count half — the prompt and the count bonus no longer double. The topScore = 1.0 half is untouched.

Worth deciding rather than patching

Options, roughly in order of how much they'd need measuring:

  1. Scale synthesized code scores into a lower band than docs, so a positional rank cannot present as a perfect match.
  2. Have the confidence heuristics ignore synthesized scores entirely and read only real relevance numbers, falling back to count when none exist.
  3. Ask whether heuristicScore should read retrieval scores at all now that groundedness assesses the actual response text.

Option 3 is the one I'd want to think about — the heuristic predates the groundedness gate, and "how good were the search results" is a weaker signal than "is the answer traceable to them", which we now compute directly.

Whichever way it goes, this is exactly what the Phase 1 harness (#241 / CPK-8076) exists to measure, so it should land after that.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with parseSnippets in pathfinder.ts and trace the synthesized scores into confidence.heuristicScore and generator.assessConfidence. Run or extend the Phase 1 harness from issue #241/CPK-8076 to measure the alternatives described, then confirm the selected confidence behavior and its effect on irrelevant code hits.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, search
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.