OpenFn / OpenFn/lightning

Allow input and output dataclips to be optional for runs

Open
#1,535 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

Most runs will have an input dataclip and an output dataclip. Right now, if either is missing from the run:start or run:complete events, Lightning will reject the event.

This is quite a serious action actually! If we reject run:complete, then obviously the run will never be marked as finished. Or if we reject run:start, then the run won't start properly, let alone finish.

There are legitimate use cases for leaf jobs to not return state. Often it doesn't make sense to return any state back to Lightning.

And I'm not entirely convinced that a job NEEDS an input state either. Maybe it calls up to a service for its input data and it just uses an upstream job as a trigger (or hell, maybe it uses a trigger with no state, like a cronjob).

In the runtime itself, I have a bug if a job doesn't return state (https://github.com/OpenFn/kit/issues/353). And I'm considering adding a crash error if a non-leaf not doesn't return state (https://github.com/OpenFn/kit/issues/449). There's also a question about error states still outstanding (https://github.com/OpenFn/kit/issues/477): if a run errors, then what state would we expect it to even return?

These questions all need working out and we need a bit of everything listed about. But I really think Lightning should be more lenient about input and output state, and I think it should be really careful about rejecting events.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing validation for the run:start and run:complete events and review the related kit issues #353, #449, and #477. Clarify the expected handling of missing input or output dataclips, including error states and non-leaf runs; done means the agreed event behavior is implemented without rejecting legitimate state-less runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.