OpenFn / OpenFn/lightning

Can we auto-retry/recover lost runs?

Open
#3,561 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

discussion
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

As a user, lost runs sometimes feel like a punishment. I didn't even do anything but the app lost my run.
Given that it is not the user's fault, is there something the app can do to auto-recover?

  • Can we auto-retry at least 3 times before calling it lost?
  • Can we auto-detect when the db is busy and possibly delay claiming of runs?
  • Suppose the worker crashes and loses runs, can we add a feature to resume the runs? The current lightning <-> worker connection is very strict, but I think if the worker can detect that it lost connection to lightning, then it can rejoin the RunChannel and continue processing.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the worker-to-Lightning RunChannel connection and the existing handling of lost runs. Compare the three proposed directions—retrying, delaying claims when the database is busy, and resuming after a worker disconnect—to determine the intended scope. Done requires an agreed recovery behavior and tests or other verification for it.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.