CopilotKit / CopilotKit/outpost

generator: fail loudly on the unusable-response modes that actually happen

Open
#231 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area: ai bug roadmap roadmap: next
Dominant language
TypeScript
Stars
7
Forks
3
Avg merge
7d 16h
Merged PRs (30d)
15

Description

From the review of #223. That PR fixed the empty-text case and pinned it; this is the higher-yield half it deliberately didn't take, plus the adjacent findings.

The main point

generate() calls messages.create with no tools and no thinking, so the API returns exactly one text block. The mode #223 guards — a tool_use or thinking block making the response read as empty — isn't reachable through this path today. Reasonable future-proofing, but the live unusable-response modes are present-but-unusable and none are guarded:

  • stop_reason === 'max_tokens' publishes a mid-sentence truncated answer as complete, at full retrieval confidence.
  • Refusals arrive as ordinary text and publish normally.

If "fail loudly on an unusable response" is the goal, stop_reason !== 'end_turn' is the check that earns it.

Not a drop-in: it changes what real reporters see on long answers, and it needs a decision about which non-end_turn reasons count as unusable (max_tokens yes; stop_sequence arguably fine). That's why it isn't in #223.

The reporter is promised a human twice

On the generation-failure path the reporter gets "A human support agent will follow up shortly" from the generator fallback and AI_DISCLAIMER_ESCALATED ("We've escalated this to our engineering team, someone will follow up in this thread shortly") from the pipeline. The suppress path has an explicit comment avoiding exactly this doubling; the generator-fallback path doesn't, and #223 routes more tickets into it.

(The escalation job is enqueued, so the promise is honored — the defect is that it's stated twice, not that it's empty.)

An empty completion permanently burns the ticket's one AI response

The throw is caught ~30 lines down, so the job returns success and never sees the retry ladder (MAX_JOB_ATTEMPTS). An empty completion is the archetypal transient failure, and the re-answer guard makes a pre-post retry safe — no BOT message exists yet at that point. Nothing is lost content-wise on this path since the text is empty by definition, so the only casualty is the retry.

A distinguishable error (EmptyModelResponseError) escaping the catch, or one in-place retry, would give the ladder a shot. The catch is the "never crash" contract every caller relies on, so this needs its own change rather than a loosened catch.

generateStream has the identical defect, unfixed

Same empty-response hole, with no fallback and no degraded signal. No production caller beyond pipeline.generateStreamingResponse. Streaming side of this is #228.

Smaller, carried from the same review

  • The throw discards real spend. The catch reports tokenUsage: { inputTokens: 0, outputTokens: 0 } but the input tokens were billed. Observability only — nothing in the queue persists tokenUsage today. The right fix is probably persisting usage at all.
  • buildSystemPrompt / buildMessages sit outside the try, so s.score.toFixed(2) on a score-less source throws past the "never crash" fallback. Same shape in generateStream.
  • generateStream yields a raw provider error as ordinary content, which the pipeline would buffer and publish — turning a 500 into a public post. No production caller, so a trap rather than a live bug. Overlaps #146.
  • The confidence clamp checks suppress only while the pipeline clamps on suppress || forcesEscalation, so confidenceLevel can read MEDIUM where the pipeline treats it as LOW.
  • Two tests inject 429/500 for a single request without maxRetries: 0, so the SDK's own retries mean they pass on mock-miss behavior rather than on the assertion.
  • The generator's degraded flag is deadpipeline.ts reads only confidenceAssessment.degraded. Pre-existing. Worth deleting or wiring, but the good outcome on the failure path comes from confidenceScore: 0, not the flag.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the generator's generate and generateStream entry points alongside pipeline.ts, then inspect the existing tests for empty responses and injected 429/500 errors. Separate the proposed response-status, fallback, retry, streaming, usage, confidence, and disclaimer behaviors into decided changes. Done means the chosen behaviors are documented by regression tests without publishing unusable responses or duplicating escalation messaging.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, backend-api-design, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.