One informative Sentry event per failed Oban job

Problem

A single failed Oban job currently produces up to three Sentry events, none of which names the worker anywhere useful:

1. Lightning.ObanManager's log blob. oban_manager.ex:19 logs a huge inspect dump starting "Oban exception:" at error level. The Sentry.LoggerHandler (added in application.ex:21-29 with capture_log_messages: true, level: :error) captures it as a message event. Every worker's failures group into one catch-all issue titled "Oban exception:" (LIGHTNING-1GA — 1,102 occurrences since 2026-03-23).
2. Lightning.ObanManager's explicit capture. oban_manager.ex:43 calls Lightning.Sentry.capture_exception/2 with worker/queue buried in extra (not faceted, not in the title) and only tags: %{type: "oban"}.
3. DBConnection's own disconnect log. When the failure is a query timeout, db_connection itself logs Postgrex.Protocol ... disconnected: client ... timed out and the LoggerHandler ships that too (LIGHTNING-1ZS — 1,104 occurrences since 2026-06-19).

During the 2026-07-01 search-vector incident, ~1,125 identical 70 s query timeouts from one worker arrived over 5.5 hours across these channels, and the only way to learn which worker was failing was to open an event and read the stacktrace. Rate limiting doesn't help — it trims volume, it never dedupes across channels.

Related defects in the same path:

• The "timeout" special case at oban_manager.ex:34 checks error.reason == :timeout, but DBConnection timeout errors carry reason: :error — real timeouts never take the "Processor Timeout" branch, and when something does match, capture_message("Processor Timeout", ...) is yet another catch-all with the worker in extra.
• The "oban-errors" telemetry handler is attached for [:oban, :circuit, :open] and [:oban, :circuit, :trip] (application.ex:58-67) but ObanManager.handle_event/4 has no clause for circuit events — if one ever fired, the FunctionClauseError would silently detach the handler. More generally, nothing in the handler is wrapped in try/rescue, so any odd error term can detach it for the life of the BEAM (this is the "quietly detaching" problem; the AI-assistant instance of it is #4780).
• handle_ai_assistant_stop/2 is dead code — the handler never subscribes to [:oban, :job, :stop].
• Both AI handlers use Repo.get! (message_processor.ex:375, :452), so a deleted ChatMessage raises Ecto.NoResultsError mid-handler.
• The AI-assistant path has its own copies of the duplication: message_processor.ex:365 logs "AI Assistant exception: ..." at error level with no logger domain (shipped by the LoggerHandler as a message event) and then captures to Sentry itself at :403/:413.

(Correction to the original write-up: sentry's own Oban.ErrorReporter is not currently subscribed — integrations.oban.capture_errors defaults to false and we don't set it. There is no upstream safety net today.)

Solution

Make sentry's built-in Oban integration the single Sentry channel for failed jobs, and stop the logger-derived duplicates at the source.

1. Enable the built-in Oban error reporter

In bootstrap.ex's config :sentry block:

config :sentry,
  ...,
  integrations: [oban: [capture_errors: true]]

Sentry.Integrations.Oban.ErrorReporter (sentry ≥ 10.3, we're on 10.9) gives us everything we're missing, maintained upstream:

• fingerprint: [job.worker, "{{ default }}"] → one Sentry issue per worker (per failure signature),
• tags: %{oban_worker: ..., oban_queue: ..., oban_state: ...} → faceted and visible at the top of the issue,
• extra with args/attempt/id/max_attempts,
• unwraps Oban.PerformError to the real exception; skips {:discard, _}/{:cancel, _} non-errors; names the worker in the message for exits/throws.

Like today's handler, it reports on every failed attempt (there is no attempt/max_attempts filter), so a job that fails three times before succeeding produces three events — all in the same per-worker issue.

2. Slim ObanManager down to what's ours

• Delete the generic clause's Sentry calls entirely: both Lightning.Sentry.capture_exception/2 and the dead-in-practice "Processor Timeout" capture_message branch (and with them the unguarded Map.get(error, :reason)).
• Replace the inspect-everything Logger.error with a one-line structured log (worker, job id, attempt, Exception.format_banner/2 of the error) so operators still see failures in application logs. Tag it with a custom logger domain, e.g. domain: [:oban_manager].
• Keep the ai_assistant clauses (MessageProcessor owns chat_messages.status recovery), but wrap each handler body in try/rescue that logs and returns :ok, so no error term can ever detach the "oban-errors" handler again.
• Strip MessageProcessor's own Sentry calls too (message_processor.ex:403, :413, and :466 if the stop handler survives): the ErrorReporter subscribes to all queues, ai_assistant included, so those captures would be duplicates under this plan. The handler keeps only the chat_messages.status recovery work. Its Logger.error at :365 gets the same domain: [:oban_manager] treatment as ObanManager's.
• Fix the attach list: drop [:oban, :circuit, :open]/[:oban, :circuit, :trip] (no clauses exist and Oban 2.x doesn't emit them) and decide [:oban, :job, :stop] — subscribe it so handle_ai_assistant_stop/2 runs, or delete the dead function.
• Swap Repo.get! for Repo.get + nil-skip in both AI handlers.

3. Keep the log-derived duplicates out of Sentry

Two small pieces of LoggerHandler hygiene in application.ex:

• Add the new domain to the handler's exclusions: excluded_domains: [:cowboy, :bandit, :oban_manager] ([:cowboy, :bandit] is the option's default — keep both when overriding). Our own failure logs stay in stdout/GCP but never become Sentry events.
• Add a before_send callback (e.g. on Lightning.SentryEventFilter) that drops message events (exception: [] on the %Sentry.Event{}) whose message.formatted matches ~r/^Postgrex\.Protocol .* disconnected/ — the raw db_connection disconnect logs. The information isn't lost: the same failure arrives as the properly-tagged Oban exception event. (Sentry.EventFilter can't do this — the filter: option is only consulted for exceptions, never message events.)

config :sentry, before_send: {Lightning.SentryEventFilter, :before_send}

One fragility worth recording: Oban.Telemetry.attach_default_logger(:debug) (application.ex:76) logs job exceptions at debug, safely below the LoggerHandler's :error threshold — but Oban's logger sets no logger domain, so if that attach level is ever raised to :error, every job failure becomes an unexcludable Sentry message event. Leave it at :debug (or add a comment saying why).

Test notes

• The Oban→Sentry path no longer goes through the Lightning.Sentry proxy, so the oban_manager_test.exs describes asserting on Lightning.MockSentry need reworking (the "other queues" block especially); the AI-assistant tests lose their capture assertions and keep the status-recovery ones.
• Sentry.Test.start_collecting_sentry_reports/0 + assert [event] = Sentry.Test.pop_sentry_reports() can assert the single-event property directly — but collection requires config :sentry, test_mode: true, which nothing sets today; add it to the test env config. Collection happens after before_send, so these assertions genuinely exercise the drop logic.

Acceptance criteria

• A failed Oban job attempt produces exactly one Sentry event — on any queue, ai_assistant included — in an issue whose title/tags identify the worker without opening a stacktrace. (Scope: Oban-infrastructure reporting. A worker that itself calls Logger.error before returning {:error, _} still adds a message event; see out of scope.)
• Postgrex.Protocol ... disconnected log messages no longer create Sentry events.
• No error term passing through the "oban-errors" handler can detach it (exercise with a nil / non-map / non-exception reason).
• A deleted ChatMessage doesn't raise in the AI handlers, and chat_messages.status recovery still runs.
• [:oban, :job, :stop] is either subscribed (and handle_ai_assistant_stop/2 runs) or the dead clause is gone; the circuit events are dropped from the attach list.
• {:discard, _}/{:cancel, _} job outcomes do not reach Sentry.
• Post-deploy: the "Oban exception:" catch-all and the Postgrex-disconnect issue stop receiving events; resolve LIGHTNING-1GA and LIGHTNING-1ZS once confirmed quiet.

Out of scope

• Workers that log at error level before failing (export_worker.ex, retry_many_workorders_job.ex, ...) still produce a LoggerHandler message event alongside the Oban one. Pre-existing behaviour; downgrading or domain-tagging those logs is a small follow-up.
• Oban cron check-ins (integrations: [oban: [cron: [enabled: true]]]) — worth a separate look; it would have flagged the search-vector worker going quiet during the 2026-07-01 incident.
• The AI SSE disconnect itself (#4748, mitigated by #4782).
• ExportWorker chunk/pool architecture (#4749).