OpenFn / OpenFn/lightning

AI assistant: oban-errors telemetry handler crashes in Sentry, silently disabling error recovery

Open
#4,780 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

AI bug elixir oban errors User Reported
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

What's broken

When an AI Assistant Oban job dies with an exit value that isn't an exception struct (a tuple like {:case_clause, ...}), the oban-errors telemetry handler updates the message status to :error correctly, then calls Lightning.Sentry.capture_exception/2 with the same value. Sentry.Event.transform_exception/2 only matches exception structs, so it raises FunctionClauseError. Because :telemetry permanently detaches a handler that raises, every subsequent AI Assistant job failure on the same pod is then ignored. The message stays at :processing forever, no broadcast fires, and the client UI sits in "..." with the input disabled.

The bug also hides itself from Sentry: the path that should report the original error is exactly the path that crashes. Only a pod restart reattaches the handler and restores recovery, until the next bad payload.

How it surfaced

A user reported AI Assistant messages stuck in "..." indefinitely on staging. The DB showed three user messages stuck in :processing across two sessions, all from CaseClauseError crashes in Tesla.Adapter.Finch.stream/3 when Apollo's SSE connection dropped mid-stream.

The first crash's exception handler ran (logs show "Updating message ... to error status after exception"). The next log line is:

[error] Handler "oban-errors" has failed and has been detached. Class=:error
Reason=:function_clause

with a stacktrace through Sentry.Event.transform_exception/2 and Sentry.capture_exception/2. The two subsequent crashes in the same window produced no exception-handler log lines at all, and their messages stayed at :processing in the DB.

Reproduced the Sentry-side crash directly on the running pod:

Sentry.Event.transform_exception({:case_clause, {:error, :fake}}, stacktrace: [])
# => FunctionClauseError: no function clause matching in Sentry.Event.transform_exception/2

What to fix

Lightning.ObanManager.handle_event/4 and MessageProcessor.handle_ai_assistant_exception/2 both pass an arbitrary error value to Lightning.Sentry.capture_exception/2. Two changes:

  • Normalize the value before handing it to Sentry. Wrap non-exception values in something Sentry.Event.transform_exception/2 can handle (e.g. a RuntimeError with inspect(reason)), or switch to Sentry.capture_message/2 when the value is not an exception struct.
  • Wrap the Sentry call in try/rescue so a Sentry crash never propagates out of the telemetry handler regardless of the input shape.

Worth adding alongside the fix as defense in depth: a sweeper that flips AI chat messages stuck in :processing beyond the configured streaming_timeout to :error, so a future regression of this kind cannot permanently strand messages.

The underlying trigger (the Tesla streaming case clause that produces these exit values) is tracked in #4781.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Lightning.ObanManager.handle_event/4 and MessageProcessor.handle_ai_assistant_exception/2, then inspect how Lightning.Sentry.capture_exception/2 handles non-exception values. Reproduce the Sentry.Event.transform_exception/2 failure shown in the issue and verify that error handling still updates the message and does not detach the telemetry handler when Sentry receives an arbitrary exit value.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.