Investigate AI Assistant SSE stream disconnects (28 in 2 days)
Nobody has claimed this yet.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
Context
A 2-day snapshot of production oban_jobs showed 28 discarded
Lightning.AiAssistant.MessageProcessor jobs, all with the same fingerprint:
** (EXIT from #PID<...>) {:case_clause,
{:error, %Mint.TransportError{reason: :closed},
{#Reference<...>, {:data, "event: <type>\ndata: \"...\""}}}}
The Finch streaming process (linked to the Oban worker) dies mid-stream when
the upstream connection closes, raising a CaseClauseError in the SSE decoder
and killing the worker via linked EXIT. MessageProcessor has max_attempts: 1
so every failure is terminal.
Code path
Lightning.ApolloClient.job_chat_stream/2(apollo_client.ex:155-178) —
Tesla +Tesla.Adapter.Finchwithopts: [adapter: [response: :stream]].AiAssistant.process_stream/3(ai_assistant.ex:1242-1274) reduces over the
lazy body stream. Itscatch :exit, reason -> ...clause does not fire
because the EXIT originates in the Finch process before the reducer sees
the chunk.
Chunk-type distribution at disconnect (sample of 26)
| Count | Chunk type | Interpretation |
|---|---|---|
| 11 | event: log with httpx line |
Very early — before model output starts |
| 10 | content_block_start/delta thinking blocks |
Mid thinking step |
| 2 | event: log adaptor fetch |
Early |
| 2 | content_block_delta text |
Real model output mid-flight |
The early-stage skew suggests infrastructure-side termination (load balancer
idle timeout, TLS reset, Apollo bridge dying) rather than user browser-close
— the user hasn't seen the first token at that point.
Investigation tasks
- Pull the 28 timestamps and search Apollo's Sentry project for matching
"client disconnected" / "upstream closed" events at the same UTC second. - Correlate with LiveView channel
terminateevents for the same users to
test the browser-close hypothesis. - For each discarded
message_id, querychat_messages.status:- If still
:processing, confirmshandle_ai_assistant_exception/2never
ran (sibling issue's root cause — handler was detached). This depends on
the handler-hardening issue landing first to be a fair test going forward. - If
:error, the exception path ran but the message is lost.
- If still
- Categorise: infra-side vs user-side vs Apollo-side.
Out of scope
No code changes in this issue. Once we know the dominant cause, follow-ups
will be filed for:
- Adding a
{:error, %Mint.TransportError{}, _}clause to the SSE decoder so
the worker stops EXITing. - Reconsidering
max_attempts: 1if the failures are transient infra issues.
Dependency
Best run after the handler-hardening issue lands — once the handler is
hardened, the data we collect will reliably indicate whether the
handler-detach pathway was masking events.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Lightning.ApolloClient.job_chat_stream/2 in apollo_client.ex:155-178 and AiAssistant.process_stream/3 in ai_assistant.ex:1242-1274, then pull the 28 timestamps. Compare Apollo Sentry events, LiveView channel terminations, and chat_messages.status for each message_id. Done means categorising the disconnects as infrastructure-, user-, or Apollo-side and recording the evidence for follow-up issues.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir
- Domain
- backend, distributed-systems, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100