CopilotKit / CopilotKit/outpost
Synthesized positional scores inflate retrieval confidence once code search returns hits
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 3
- Avg merge
- 7d 16h
- Merged PRs (30d)
- 15
Description
Found by the review of PR #242 (Phase 2, code search). Not fixed there because it changes confidence behaviour on every ticket and wants measuring rather than guessing.
parseSnippets synthesizes a positional score for text-format results — score: Math.max(0.5, 1 - i * 0.05) (pathfinder.ts) — because the MCP text format carries no numeric relevance. That was tolerable while only search-docs produced them. Now search-code produces its own independent list, so the top code hit gets 1.0 regardless of how marginal it is, and min_score defaults to 0.3.
Two consumers read those numbers:
confidence.heuristicScore—topScore * 0.4 + avgScore * 0.4 + min(count * 0.03, 0.2)generator.assessConfidence—avg + min(count * 0.05, 0.15)
So retrieval confidence is inflated for any question that returns any code hit, including irrelevant ones. That is the opposite of what the groundedness work in this module was for.
PR #242 caps the merged list at defaultLimit, which fixes the count half — the prompt and the count bonus no longer double. The topScore = 1.0 half is untouched.
Worth deciding rather than patching
Options, roughly in order of how much they'd need measuring:
- Scale synthesized code scores into a lower band than docs, so a positional rank cannot present as a perfect match.
- Have the confidence heuristics ignore synthesized scores entirely and read only real relevance numbers, falling back to count when none exist.
- Ask whether
heuristicScoreshould read retrieval scores at all now that groundedness assesses the actual response text.
Option 3 is the one I'd want to think about — the heuristic predates the groundedness gate, and "how good were the search results" is a weaker signal than "is the answer traceable to them", which we now compute directly.
Whichever way it goes, this is exactly what the Phase 1 harness (#241 / CPK-8076) exists to measure, so it should land after that.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with parseSnippets in pathfinder.ts and trace the synthesized scores into confidence.heuristicScore and generator.assessConfidence. Run or extend the Phase 1 harness from issue #241/CPK-8076 to measure the alternatives described, then confirm the selected confidence behavior and its effect on irrelevant code hits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100