Allow input and output dataclips to be optional for runs
Nobody has claimed this yet.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
Most runs will have an input dataclip and an output dataclip. Right now, if either is missing from the run:start or run:complete events, Lightning will reject the event.
This is quite a serious action actually! If we reject run:complete, then obviously the run will never be marked as finished. Or if we reject run:start, then the run won't start properly, let alone finish.
There are legitimate use cases for leaf jobs to not return state. Often it doesn't make sense to return any state back to Lightning.
And I'm not entirely convinced that a job NEEDS an input state either. Maybe it calls up to a service for its input data and it just uses an upstream job as a trigger (or hell, maybe it uses a trigger with no state, like a cronjob).
In the runtime itself, I have a bug if a job doesn't return state (https://github.com/OpenFn/kit/issues/353). And I'm considering adding a crash error if a non-leaf not doesn't return state (https://github.com/OpenFn/kit/issues/449). There's also a question about error states still outstanding (https://github.com/OpenFn/kit/issues/477): if a run errors, then what state would we expect it to even return?
These questions all need working out and we need a bit of everything listed about. But I really think Lightning should be more lenient about input and output state, and I think it should be really careful about rejecting events.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing validation for the run:start and run:complete events and review the related kit issues #353, #449, and #477. Clarify the expected handling of missing input or output dataclips, including error states and non-leaf runs; done means the agreed event behavior is implemented without rejecting legitimate state-less runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100