OpenFn / OpenFn/kit

Worker: when `workflow-complete` is rejected, try to send a `workfkow-error`

Open
#1,218 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

DevX lost
Dominant language
TypeScript
Stars
21
Forks
23
Avg merge
1d 22h
Merged PRs (30d)
17

Description

Sometimes the worker can fail to send workflow-complete events, resulting in a run being marked Lost. This is very rare but it does happen. Lately we've had a few database timeouts which trigger it (possibly while uploading large dataclips?)

But the point for Lost runs is: this shouldn't be lost. The worker KNOWs that the complete event was rejected.

A good strategy in this case would be:

  1. Retry the event
  2. Retry the event without a dataclip
  3. Send an error back to lightning

In the event of an error, we should retry indefinitely.

Obviously there's a problem that if the system is under extreme load, events may not get processed. But dropping the dataclip and retrying should get it go through

(btw I forget how the error event works - do we actually send this to lightning or does and error just generate a complete with an exit reason?)

(also why do we send a final dataclip? Should't we just send the id? Didn't we make a change here recently)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the worker's handling of rejected workflow-complete events and read how error events are sent to Lightning. Check the existing dataclip and retry behavior. Done means a rejected completion follows the intended retry, dataclip fallback, and error-reporting path instead of leaving the run marked Lost.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.