OpenFn / OpenFn/lightning

Investigate "RunChannel replied with error" (LIGHTNING-WG)

Open
#4,794 0 comments 0 reactions 1 assignee View on GitHub

@stuartc is already working on this.

Since May 26, 2026.

bug channels lost runs needs investigation Sentry
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

Summary

The Sentry issue tagged LIGHTNING-WG ("RunChannel replied with
error") is a catch-all bucket: the Sentry message string is
hard-coded in lib/lightning_web/channels/channel_helpers.ex, so every
{:error, _} reply from the run channel collapses into one issue
regardless of root cause.

Inspecting extra.error reveals at least three distinct failure modes
mixed under this single bucket. Some appear to be benign noise (run
still completes); at least one appears to correspond to real
completion loss (run reaped as :lost with no final_dataclip
persisted).

Cause is not yet localised — it could be Lightning-side (channel /
completion contract), worker-side (event ordering / pod behaviour),
or an interaction between the two.

Why we care

  • Sentry noise is masking a real signal — we can't tell shape-from-shape
    without inspecting extra.error on each event.
  • One of the shapes appears to correlate with runs ending :lost
    without a final dataclip.
  • Hard to triage further without server-side differentiation.

Scope

Investigation only. No fix proposed. We need to:

  1. Confirm or refute the per-shape root-cause theories.
  2. Decide what to fix where (Lightning, ws-worker, or both).
  3. Improve observability so future occurrences are self-classifying.

Error shapes observed

  • Shape Afinal_dataclip_id: ["does not exist"]. Worker
    reports a final dataclip Lightning can't find.
  • Shape Bstep_id: ["must be associated with the run"].
    Log line / event references a step_id not tied to this run.
  • Shape Cstate: ["already in completed state"]. Completion
    attempt against a run Lightning has already terminated.

Proposed next steps

  • Split the Sentry capture per shape (or add a shape tag) so
    future events are self-classifying.
  • Per-shape investigation: confirm/refute root cause for A, B, C
    independently.
  • Decide on Lightning-side observability improvements (e.g. attach
    worker identity to channel messages) to make duplicate-claim or
    cross-wire scenarios trivially detectable.
  • If/when a worker-side cause is confirmed, open a corresponding
    issue on OpenFn/kit.

Not in scope

  • Silencing the Sentry capture. Multiple distinct contract violations
    appear to be hidden under one bucket; muting would lose the signal
    entirely.

References

  • Related historical PRs: #4531 (introduces final_dataclip_id /
    final_state on run:complete), #4626 (non-map final_state fix).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.