AI assistant: oban-errors telemetry handler crashes in Sentry, silently disabling error recovery
Nobody has claimed this yet.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
What's broken
When an AI Assistant Oban job dies with an exit value that isn't an exception struct (a tuple like {:case_clause, ...}), the oban-errors telemetry handler updates the message status to :error correctly, then calls Lightning.Sentry.capture_exception/2 with the same value. Sentry.Event.transform_exception/2 only matches exception structs, so it raises FunctionClauseError. Because :telemetry permanently detaches a handler that raises, every subsequent AI Assistant job failure on the same pod is then ignored. The message stays at :processing forever, no broadcast fires, and the client UI sits in "..." with the input disabled.
The bug also hides itself from Sentry: the path that should report the original error is exactly the path that crashes. Only a pod restart reattaches the handler and restores recovery, until the next bad payload.
How it surfaced
A user reported AI Assistant messages stuck in "..." indefinitely on staging. The DB showed three user messages stuck in :processing across two sessions, all from CaseClauseError crashes in Tesla.Adapter.Finch.stream/3 when Apollo's SSE connection dropped mid-stream.
The first crash's exception handler ran (logs show "Updating message ... to error status after exception"). The next log line is:
[error] Handler "oban-errors" has failed and has been detached. Class=:error
Reason=:function_clause
with a stacktrace through Sentry.Event.transform_exception/2 and Sentry.capture_exception/2. The two subsequent crashes in the same window produced no exception-handler log lines at all, and their messages stayed at :processing in the DB.
Reproduced the Sentry-side crash directly on the running pod:
Sentry.Event.transform_exception({:case_clause, {:error, :fake}}, stacktrace: [])
# => FunctionClauseError: no function clause matching in Sentry.Event.transform_exception/2
What to fix
Lightning.ObanManager.handle_event/4 and MessageProcessor.handle_ai_assistant_exception/2 both pass an arbitrary error value to Lightning.Sentry.capture_exception/2. Two changes:
- Normalize the value before handing it to Sentry. Wrap non-exception values in something
Sentry.Event.transform_exception/2can handle (e.g. aRuntimeErrorwithinspect(reason)), or switch toSentry.capture_message/2when the value is not an exception struct. - Wrap the Sentry call in
try/rescueso a Sentry crash never propagates out of the telemetry handler regardless of the input shape.
Worth adding alongside the fix as defense in depth: a sweeper that flips AI chat messages stuck in :processing beyond the configured streaming_timeout to :error, so a future regression of this kind cannot permanently strand messages.
The underlying trigger (the Tesla streaming case clause that produces these exit values) is tracked in #4781.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Lightning.ObanManager.handle_event/4 and MessageProcessor.handle_ai_assistant_exception/2, then inspect how Lightning.Sentry.capture_exception/2 handles non-exception values. Reproduce the Sentry.Event.transform_exception/2 failure shown in the issue and verify that error handling still updates the message and does not detach the telemetry handler when Sentry receives an arbitrary exit value.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100