CopilotKit / CopilotKit/outpost
groundedness: gaps and pre-existing defects surfaced by the #147 review
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 3
- Avg merge
- 7d 16h
- Merged PRs (30d)
- 15
Description
Collected from the review of #222 so they don't get lost. None are introduced by that PR; it fixed its own blocker plus the two test-quality findings. Grouped by whether acting on them widens the gate (risks withholding correct answers) or narrows it.
Reminder of the stakes: after #143 the identifier signal is the only mechanism that withholds a response. A false positive here withholds a correct answer from a real reporter.
A. Spellings that escape the gate (widening — needs a false-positive measurement pass)
All verified returning []:
- Undotted kebab —
`copilot-ghost-panel`. The dotted form is covered; the bare form isn't. - camelCase invented names —
`copilotFabricate()`,`copilotInvented`. - Lowercase after the prefix —
`useCopilotfabricated()`,`<Copilotinvented />`. Plausible model typos. - Casing asymmetry worth at least a doc line:
Copilot_Chatis an identifier because_is in the character class, butcopilot_chatisn't;COPILOT_API_KEYisn't whileCOPILOTKIT_TOKENis.
Each of these widens the gate, so each carries false-positive risk. They want one deliberate pass with a corpus, not four separate regex tweaks. The strings above are the starting test matrix.
B. The retrieval-quality trade (decide, don't drift)
The widened gate can't distinguish a real API name from an invented one, so its false-positive rate is bounded entirely by retrieval quality. Measured: Register it with `useCopilotAction()` and mount `<CopilotSidebar />`. against a retrieved CopilotChat page not literally containing those strings → suppress = true.
So the five-ish names a correct answer cites most often are the ones most likely to be withheld. An allowlist of those five was considered and rejected in #222 — it doesn't help the #147 case (the fabrication isn't on the list either), and it converts "withheld a correct answer" into "published an invented one".
The actual fix is retrieval: if CopilotChat's page doesn't contain useCopilotAction, an answer citing both is standing on a source that doesn't support it. Tracking that here rather than papering over it in the scorer.
C. Pre-existing defects (narrowing or correctness — safe to fix)
IDENTIFIER_PATHaccepts a leading#but the split strips only.—`#copilotKitPanel`may be recorded with the#attached and dedup separately from its dotted form, turning one invented name into two entries and clearing the suppression bar alone. Same root cause means ID selectors can never ground.BARE_HOST_PATTERN's TLD list includesapp|dev|io|co|sh— so`copilotKitBridge.io`is erased as a hostname before extraction sees it.- The "never throws" docblock is wrong — a null
sources, or a null element in the array, throws a TypeError inside the publish/withhold decision. - The hedge reason quotes total
hedgeCountwhile onlyexcessHedgeswas charged. - Backticked content over 80 chars is silently skipped — undocumented and untested.
s.sourceUrlis in the grounding haystack, so any identifier that is a substring ofdocs.copilotkit.aigrounds automatically. Believed deliberate; needs a comment either way so the next reader doesn't re-flag it.
D. Test-quality (two more found, both may pass vacuously)
- The "claim wording never withholds" assertion appears wrapped in an
if— which would make it pass vacuously the moment extraction starts inventing identifiers, i.e. exactly the regression it exists to catch. - The name-claims test passes
CHAT_DOCSand asserts neither coverage nor empty sources.
Two others from the same review — the non-discriminating "English word" corpus row and the unpinned ^ anchors — were fixed in #222.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the groundedness extraction and publish/withhold decision, then inspect the existing name-claims and claim-wording tests mentioned in the issue. Use the listed identifier strings as the regression matrix and verify the null-input, selector, hostname, hedge-count, and long-backtick cases. Done requires agreed scope, non-vacuous tests, safe defect fixes, and a documented retrieval-quality decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100