OpenFn / OpenFn/lightning

Carry workflow state in the wire format (project.yaml + CLI)

Open
#4,897 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

blocked:needs-decision phase:2-later Sandboxes DevX
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

As a developer deploying workflows through the CLI, GitHub sync, or YAML, I want a workflow's lifecycle state to travel in the wire format, so that exporting and re-importing a live workflow keeps it live instead of quietly reverting to a draft.

Part of #4852 (Sandbox DevX epic). Cross-repo. Blocked on the CLI and provisioner push rules (see below).

What's broken

The workflow lifecycle field state (draft | live) is the user-facing contract for whether a workflow is processing data, but it does not travel on the wire. The provision API JSON, the project.yaml export, and GitHub sync all round-trip trigger.enabled as a first-class field, while none of them carry workflow.state. The CLI (in the kit repo) round-trips and hashes trigger.enabled for its diff engine and has no lifecycle concept at all.

How it surfaced

While completing the lifecycle epic we found round-trips are lossy for lifecycle: export a live workflow, re-import it via CLI deploy, GitHub sync, or YAML import, and it comes back as a draft because state is absent and resets to the schema default. As an interim, non-breaking fix, the importer now infers state from the resulting trigger-enabled set (any enabled trigger means live, otherwise draft), with an explicit state in the spec always winning. That keeps imports coherent, but it is an inference, not the real contract.

Verified safe to defer, unsafe to half-build (2026-07-05)

Checked against prod and both repos. Phase 1 ships with zero wire-format regression WITHOUT this issue. The Lightning export never emits state (export code is identical on main and the epic branch) and still emits trigger.enabled, and import is additive (infers state from enabled when absent). A prod check found no workflow with mixed enabled/disabled triggers (0 of 2,857 active), so the inference is exact for every workflow and there are no phantom CLI diffs.

The danger is doing the EXPORT half of this issue first. If Lightning starts emitting state before lexicon and the CLI accept it:

  • An older or not-yet-upgraded provisioner rejects any document containing state (validate_extraneous_params returns a 422), breaking GitHub sync and CLI deploy during rollout and against older instances.
  • Both CLI paths silently cache and round-trip an unknown state back on the next deploy, and the epic-branch provisioner applies an explicit incoming state (explicit wins over inference), so a stale cached value can silently flip a workflow's draft/live state on an unrelated deploy. That is silent data loss.

So the inference fallback is not just interim, it is the correct behaviour until the whole chain is ready.

Required build order

Consumers before producers: (1) lexicon adds state to the workflow spec; (2) kit round-trips state and drops trigger.enabled from the change-hash in favour of state (a server-derived enabled would otherwise phantom-diff); (3) only then Lightning adds state to the export ordering map and provisioning JSON, keeping the absent-state inference as the fallback for older CLIs. Lightning must not emit state in the export until steps 1 and 2 ship.

What to fix

Make state an explicit part of the wire format. This is a coordinated change across repos:

  • lexicon: add state to the workflow spec schema.
  • kit (packages/project, packages/deploy): serialize and round-trip workflow.state, and decide whether trigger.enabled stays in the trigger change-hash. With server-side coherence, a server-derived enabled flowing back through the change-hash produces phantom diffs, so enabled likely needs to drop out of the hash in favour of state.
  • Lightning: add state to the export ordering map and the provisioning JSON, cast it in the provisioner workflow changeset (keeping the absent-state inference as the fallback for legacy specs and older CLIs), and derive enabled from the resolved state on import.

Blocked on

The CLI and provisioner push rules from the Jun 24 planning call are undefined: on a push to a live or protected project, do we reject, create a draft, update a draft, or error if none exists, and what happens to sibling workflows on a single-workflow deploy. Until those rules and the protected-project decision are settled, the inference fallback is the agreed behaviour. Labeled blocked:needs-decision.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the workflow spec in lexicon, then inspect kit's packages/project and packages/deploy round-trip and hash logic. Review Lightning's export ordering map and provisioning JSON, but first resolve the documented CLI and provisioner push rules. Done means all three repositories accept, preserve, and safely import workflow state without premature export changes or legacy inference regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir, github, json, yaml
Domain
api, backend, cli, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.