CopilotKit / CopilotKit/outpost
Groundedness gate: streaming path, generator level, and QA metadata still read the pre-#143 signal
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 3
- Avg merge
- 7d 16h
- Merged PRs (30d)
- 15
Description
The groundedness gate is enforced on some paths and signals but not others. Three consumers still read a signal whose meaning changed in #143.
1. generateStreamingResponse gates on suppress only
packages/outpost/ai/src/pipeline.ts (streaming path) applies the suppression swap and nothing else — no groundedness penalty, no charged-claim clamp, no calibration, no formatter, no disclaimer, and no escalation.
So identical text takes two different routes:
| non-streaming | streaming | |
|---|---|---|
| "Bug confirmed. Root cause is X." (no invented identifiers) | penalty 0.35, clamped below ESCALATE, disclaimer added, human paged | streams unmarked, no disclaimer, nobody paged |
This was covered incidentally while suppression was claim-driven. #143 moved claims to penalty-only and the streaming path kept checking suppress.
Not currently reachable in production — generateStreamingResponse has no production caller (apps/web/src/app/api/qa/route.ts uses the non-streaming path and chunks the finished text itself). It is a trap for the next caller, not a live bug.
2. The generator's confidenceLevel omits the claim clamp
packages/outpost/ai/src/generator.ts → classifyGroundedConfidence clamps to SUPPRESSED_CONFIDENCE_CAP when groundedness.suppress is set, but does not clamp on groundedness.unverifiedClaims.length > 0. pipeline.ts does both. So for a charged-claim response the two levels disagree — the generator can report a level the pipeline would never report for the same text.
3. The QA route's metadata event omits suppressed
apps/web/src/app/api/qa/route.ts streams the safe published text (correct — it inherits the gate), but its metadata SSE event carries confidence, sources and latencyMs only. The UI cannot distinguish "here is your answer" from "we withheld an ungrounded draft and escalated", so both render identically.
Root cause, and the fix worth making
These are three instances of one process failure across #143: a signal's meaning changed and not every reader was swept. Same shape as the autoSend bug fixed earlier in the same PR (penalty moved to the pipeline, autoSend kept reading the pre-penalty score).
Patching the three sites invites a fourth. The structural fix is a single parameterized test asserting the gate's contract holds on every path and consumer — non-streaming pipeline, streaming pipeline, queue handler, QA route — so adding a path or changing what a signal means fails loudly instead of silently. Additive, and it would have caught all three of these plus autoSend.
Found by the CR loop on #143 (rounds 2-3), each reproduced against the merged code.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the groundedness gate through packages/outpost/ai/src/pipeline.ts and generator.ts, then inspect apps/web/src/app/api/qa/route.ts and the queue handler. Compare each consumer against the contract described in the issue and locate the existing gate tests. Done means all listed paths expose consistent groundedness behavior and QA metadata, with parameterized coverage preventing future signal-reader drift.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, api, backend, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100