OpenFn / OpenFn/lightning

Investigate AI Assistant SSE stream disconnects (28 in 2 days)

Open
#4,748 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

AI needs investigation oban errors
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

Context

A 2-day snapshot of production oban_jobs showed 28 discarded
Lightning.AiAssistant.MessageProcessor jobs, all with the same fingerprint:

** (EXIT from #PID<...>) {:case_clause,
  {:error, %Mint.TransportError{reason: :closed},
   {#Reference<...>, {:data, "event: <type>\ndata: \"...\""}}}}

The Finch streaming process (linked to the Oban worker) dies mid-stream when
the upstream connection closes, raising a CaseClauseError in the SSE decoder
and killing the worker via linked EXIT. MessageProcessor has max_attempts: 1
so every failure is terminal.

Code path

  • Lightning.ApolloClient.job_chat_stream/2 (apollo_client.ex:155-178) —
    Tesla + Tesla.Adapter.Finch with opts: [adapter: [response: :stream]].
  • AiAssistant.process_stream/3 (ai_assistant.ex:1242-1274) reduces over the
    lazy body stream. Its catch :exit, reason -> ... clause does not fire
    because the EXIT originates in the Finch process before the reducer sees
    the chunk.

Chunk-type distribution at disconnect (sample of 26)

Count Chunk type Interpretation
11 event: log with httpx line Very early — before model output starts
10 content_block_start/delta thinking blocks Mid thinking step
2 event: log adaptor fetch Early
2 content_block_delta text Real model output mid-flight

The early-stage skew suggests infrastructure-side termination (load balancer
idle timeout, TLS reset, Apollo bridge dying) rather than user browser-close
— the user hasn't seen the first token at that point.

Investigation tasks

  • Pull the 28 timestamps and search Apollo's Sentry project for matching
    "client disconnected" / "upstream closed" events at the same UTC second.
  • Correlate with LiveView channel terminate events for the same users to
    test the browser-close hypothesis.
  • For each discarded message_id, query chat_messages.status:
    • If still :processing, confirms handle_ai_assistant_exception/2 never
      ran (sibling issue's root cause — handler was detached). This depends on
      the handler-hardening issue landing first to be a fair test going forward.
    • If :error, the exception path ran but the message is lost.
  • Categorise: infra-side vs user-side vs Apollo-side.

Out of scope

No code changes in this issue. Once we know the dominant cause, follow-ups
will be filed for:

  • Adding a {:error, %Mint.TransportError{}, _} clause to the SSE decoder so
    the worker stops EXITing.
  • Reconsidering max_attempts: 1 if the failures are transient infra issues.

Dependency

Best run after the handler-hardening issue lands — once the handler is
hardened, the data we collect will reliably indicate whether the
handler-detach pathway was masking events.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Lightning.ApolloClient.job_chat_stream/2 in apollo_client.ex:155-178 and AiAssistant.process_stream/3 in ai_assistant.ex:1242-1274, then pull the 28 timestamps. Compare Apollo Sentry events, LiveView channel terminations, and chat_messages.status for each message_id. Done means categorising the disconnects as infrastructure-, user-, or Apollo-side and recording the evidence for follow-up issues.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
backend, distributed-systems, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.