posit-dev / posit-dev/shinychat
Surface non-standard LLM finish reasons (max_tokens, content_filter, etc.) as UI notifications, not just console warnings
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 139
- Forks
- 28
- Avg merge
- 23h 44m
- Merged PRs (30d)
- 50
Description
Problem
When a model stops for a reason other than a clean completion — hitting max_tokens, exceeding the context window, getting filtered by content moderation — ellmer and chatlas already detect this (AssistantTurn.finish_reason / check_finish_reason()) and warn about it. But that warning is console-only. In a Shiny Chat app, it's invisible: the user just sees whatever failure happens downstream (e.g. a confusing JSON parse error on truncated output), with no clue that the actual cause was a token limit.
This came up concretely in querychat: a truncated response caused a parse error: premature EOF in the chat UI, while the real cause — "Response was truncated because it hit the max_tokens limit" — only ever printed to the R console.
Proposed direction
- ellmer and chatlas each upgrade their existing plain warning (
cli::cli_warn()/warnings.warn(..., UserWarning)) to a classed/typed signal that carries the finish reason as data — anellmer_finish_reasonrlang condition class in R, anIncompleteResponseWarningsubclass in Python (chatlas already has this exact pattern for a different case,ToolFailureWarning). - shinychat's R (
chat_append_stream()) and Python (append_message_stream()) sides each catch that signal while consuming the stream and surface it as a UI notification (showNotification()/notification_show()), matching thetype="warning"treatment each side already uses elsewhere for errors. - No public API signature changes anticipated on either side.
A full design (data flow, exact call sites, edge cases, one open technical assumption still needing a smoke test) is written up and available on request — happy to paste it in or link it once it has a stable home.
Scope
This covers ellmer, chatlas, and shinychat (R + Python). It does not cover querychat's own downstream JSON-parsing error handling, which is a separate, narrower fix.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read the existing finish-reason handling around AssistantTurn.finish_reason and check_finish_reason(), then inspect shinychat's chat_append_stream() and append_message_stream() consumers. Verify the proposed signal paths and the open smoke-test assumption; done means non-standard finish reasons appear as warning notifications in both R and Python streaming UIs without public API signature changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, r
- Domain
- backend, frontend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100