CopilotKit / CopilotKit/outpost

Groundedness gate: streaming path, generator level, and QA metadata still read the pre-#143 signal

Open
#146 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: ai roadmap roadmap: next
Dominant language
TypeScript
Stars
7
Forks
3
Avg merge
7d 16h
Merged PRs (30d)
15

Description

The groundedness gate is enforced on some paths and signals but not others. Three consumers still read a signal whose meaning changed in #143.

1. generateStreamingResponse gates on suppress only

packages/outpost/ai/src/pipeline.ts (streaming path) applies the suppression swap and nothing else — no groundedness penalty, no charged-claim clamp, no calibration, no formatter, no disclaimer, and no escalation.

So identical text takes two different routes:

non-streaming streaming
"Bug confirmed. Root cause is X." (no invented identifiers) penalty 0.35, clamped below ESCALATE, disclaimer added, human paged streams unmarked, no disclaimer, nobody paged

This was covered incidentally while suppression was claim-driven. #143 moved claims to penalty-only and the streaming path kept checking suppress.

Not currently reachable in productiongenerateStreamingResponse has no production caller (apps/web/src/app/api/qa/route.ts uses the non-streaming path and chunks the finished text itself). It is a trap for the next caller, not a live bug.

2. The generator's confidenceLevel omits the claim clamp

packages/outpost/ai/src/generator.tsclassifyGroundedConfidence clamps to SUPPRESSED_CONFIDENCE_CAP when groundedness.suppress is set, but does not clamp on groundedness.unverifiedClaims.length > 0. pipeline.ts does both. So for a charged-claim response the two levels disagree — the generator can report a level the pipeline would never report for the same text.

3. The QA route's metadata event omits suppressed

apps/web/src/app/api/qa/route.ts streams the safe published text (correct — it inherits the gate), but its metadata SSE event carries confidence, sources and latencyMs only. The UI cannot distinguish "here is your answer" from "we withheld an ungrounded draft and escalated", so both render identically.

Root cause, and the fix worth making

These are three instances of one process failure across #143: a signal's meaning changed and not every reader was swept. Same shape as the autoSend bug fixed earlier in the same PR (penalty moved to the pipeline, autoSend kept reading the pre-penalty score).

Patching the three sites invites a fourth. The structural fix is a single parameterized test asserting the gate's contract holds on every path and consumer — non-streaming pipeline, streaming pipeline, queue handler, QA route — so adding a path or changing what a signal means fails loudly instead of silently. Additive, and it would have caught all three of these plus autoSend.

Found by the CR loop on #143 (rounds 2-3), each reproduced against the merged code.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the groundedness gate through packages/outpost/ai/src/pipeline.ts and generator.ts, then inspect apps/web/src/app/api/qa/route.ts and the queue handler. Compare each consumer against the contract described in the issue and locate the existing gate tests. Done means all listed paths expose consistent groundedness behavior and QA metadata, with parameterized coverage preventing future signal-reader drift.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, api, backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.