Investigate "RunChannel replied with error" (LIGHTNING-WG)
@stuartc is already working on this.
Since May 26, 2026.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
Summary
The Sentry issue tagged LIGHTNING-WG ("RunChannel replied with
error") is a catch-all bucket: the Sentry message string is
hard-coded in lib/lightning_web/channels/channel_helpers.ex, so every
{:error, _} reply from the run channel collapses into one issue
regardless of root cause.
Inspecting extra.error reveals at least three distinct failure modes
mixed under this single bucket. Some appear to be benign noise (run
still completes); at least one appears to correspond to real
completion loss (run reaped as :lost with no final_dataclip
persisted).
Cause is not yet localised — it could be Lightning-side (channel /
completion contract), worker-side (event ordering / pod behaviour),
or an interaction between the two.
Why we care
- Sentry noise is masking a real signal — we can't tell shape-from-shape
without inspectingextra.erroron each event. - One of the shapes appears to correlate with runs ending
:lost
without a final dataclip. - Hard to triage further without server-side differentiation.
Scope
Investigation only. No fix proposed. We need to:
- Confirm or refute the per-shape root-cause theories.
- Decide what to fix where (Lightning, ws-worker, or both).
- Improve observability so future occurrences are self-classifying.
Error shapes observed
- Shape A —
final_dataclip_id: ["does not exist"]. Worker
reports a final dataclip Lightning can't find. - Shape B —
step_id: ["must be associated with the run"].
Log line / event references a step_id not tied to this run. - Shape C —
state: ["already in completed state"]. Completion
attempt against a run Lightning has already terminated.
Proposed next steps
- Split the Sentry capture per shape (or add a
shapetag) so
future events are self-classifying. - Per-shape investigation: confirm/refute root cause for A, B, C
independently. - Decide on Lightning-side observability improvements (e.g. attach
worker identity to channel messages) to make duplicate-claim or
cross-wire scenarios trivially detectable. - If/when a worker-side cause is confirmed, open a corresponding
issue onOpenFn/kit.
Not in scope
- Silencing the Sentry capture. Multiple distinct contract violations
appear to be hidden under one bucket; muting would lose the signal
entirely.
References
- Related historical PRs: #4531 (introduces
final_dataclip_id/
final_stateonrun:complete), #4626 (non-mapfinal_statefix).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.