Worker: when `workflow-complete` is rejected, try to send a `workfkow-error`
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 21
- Forks
- 23
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 17
Description
Sometimes the worker can fail to send workflow-complete events, resulting in a run being marked Lost. This is very rare but it does happen. Lately we've had a few database timeouts which trigger it (possibly while uploading large dataclips?)
But the point for Lost runs is: this shouldn't be lost. The worker KNOWs that the complete event was rejected.
A good strategy in this case would be:
- Retry the event
- Retry the event without a dataclip
- Send an error back to lightning
In the event of an error, we should retry indefinitely.
Obviously there's a problem that if the system is under extreme load, events may not get processed. But dropping the dataclip and retrying should get it go through
(btw I forget how the error event works - do we actually send this to lightning or does and error just generate a complete with an exit reason?)
(also why do we send a final dataclip? Should't we just send the id? Didn't we make a change here recently)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the worker's handling of rejected workflow-complete events and read how error events are sent to Lightning. Check the existing dataclip and retry behavior. Done means a rejected completion follows the intended retry, dataclip fallback, and error-reporting path instead of leaving the run marked Lost.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100