CopilotKit / CopilotKit/outpost
Withheld drafts are indistinguishable from delivered replies (replayed as assistant history, no DB marker)
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 3
- Avg merge
- 7d 16h
- Merged PRs (30d)
- 15
Description
When the groundedness gate withholds an ungrounded draft, the reporter gets safe replacement copy and a human is paged (#143). But nothing in the database records that the draft was withheld, and two downstream readers treat it as a delivered reply.
1. The withheld draft re-enters the model's context as an assistant turn
packages/outpost/queue/src/handlers/ai-response.ts persists pipelineResult.response — the raw draft — as a type: 'BOT' message. That is intentional: the human handling the escalation needs the draft to work from.
The consequence is not intentional. The same handler builds conversationHistory from prior messages, mapping BOT rows to assistant turns. So on the ticket's next AI response, the fabrication the reporter never saw is fed back to the model as something it apparently already said — and the model has no way to know it was withheld. A "bug confirmed" draft that was suppressed becomes prior context asserting the bug is confirmed.
2. suggestedResponse drops everything past the first Discord chunk
Same handler stores pipelineResult.formatted.text into ticket.suggestedResponse. For a long Discord reply, formatted.text is only parts[0] — the remaining parts live in formatted.parts. The human's editor and the shadow-mode record therefore get a truncated reply with no indication anything was dropped.
3. No column marks a message as withheld
Message carries confidenceScore and confidenceLevel; nothing records suppressed or the groundedness reasons. Suppression exists only in worker logs and the escalation job's reason string. So the dashboard cannot show "this draft was withheld for fabrication", analytics cannot count how often the gate fires, and #145 (the reply editor prefilling the disclaimer) cannot be fixed cleanly without it.
Suggested shape
- Persist the suppression verdict — a
suppressedboolean and the groundedness reasons onMessage, which needs a Prisma migration. - Exclude withheld drafts from
conversationHistoryconstruction, or mark them so they are excluded. - Use
formatted.partswhere present rather than assumingtextis the whole reply.
Relates to #145 (reply editor prefills the AI disclaimer instead of the draft) — both are consequences of suggestedResponse carrying published copy while response carries the draft, with nothing distinguishing the two states.
Found by the CR loop on #143 (round 3), four reviewers independently.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in packages/outpost/queue/src/handlers/ai-response.ts and trace how pipelineResult.response, formatted.parts, and conversationHistory are persisted or built. Review the Message schema and add the Prisma migration described in the issue. Done means withheld drafts are marked with their reasons, excluded from assistant history, and suggestedResponse preserves all formatted parts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, backend, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100