posit-dev / posit-dev/shinychat

Surface non-standard LLM finish reasons (max_tokens, content_filter, etc.) as UI notifications, not just console warnings

Open
#408 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ai-triage:done Priority: Medium
Dominant language
TypeScript
Stars
139
Forks
28
Avg merge
23h 44m
Merged PRs (30d)
50

Description

Problem

When a model stops for a reason other than a clean completion — hitting max_tokens, exceeding the context window, getting filtered by content moderation — ellmer and chatlas already detect this (AssistantTurn.finish_reason / check_finish_reason()) and warn about it. But that warning is console-only. In a Shiny Chat app, it's invisible: the user just sees whatever failure happens downstream (e.g. a confusing JSON parse error on truncated output), with no clue that the actual cause was a token limit.

This came up concretely in querychat: a truncated response caused a parse error: premature EOF in the chat UI, while the real cause — "Response was truncated because it hit the max_tokens limit" — only ever printed to the R console.

Proposed direction
  • ellmer and chatlas each upgrade their existing plain warning (cli::cli_warn() / warnings.warn(..., UserWarning)) to a classed/typed signal that carries the finish reason as data — an ellmer_finish_reason rlang condition class in R, an IncompleteResponseWarning subclass in Python (chatlas already has this exact pattern for a different case, ToolFailureWarning).
  • shinychat's R (chat_append_stream()) and Python (append_message_stream()) sides each catch that signal while consuming the stream and surface it as a UI notification (showNotification() / notification_show()), matching the type="warning" treatment each side already uses elsewhere for errors.
  • No public API signature changes anticipated on either side.

A full design (data flow, exact call sites, edge cases, one open technical assumption still needing a smoke test) is written up and available on request — happy to paste it in or link it once it has a stable home.

Scope

This covers ellmer, chatlas, and shinychat (R + Python). It does not cover querychat's own downstream JSON-parsing error handling, which is a separate, narrower fix.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the existing finish-reason handling around AssistantTurn.finish_reason and check_finish_reason(), then inspect shinychat's chat_append_stream() and append_message_stream() consumers. Verify the proposed signal paths and the open smoke-test assumption; done means non-standard finish reasons appear as warning notifications in both R and Python streaming UIs without public API signature changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, r
Domain
backend, frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.