Add full kit-lightning integration test suite
@doc-han is already working on this.
Since Jul 2, 2026.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
The problem
Lightning and kit (worker + CLI) ship independently, but they share a contract - provisioner API, deploy v1/v2, the worker WebSocket protocol - that nothing currently exercises end-to-end against the heads of both repos.
Lightning's integration suite runs against a pinned @openfn/ws-worker and @openfn/cli. Kit's suite uses the in-repo lightning-mock. Both are useful, neither catches the case where a new worker changes a behaviour that real Lightning depends on.
The recent example: worker #1306 trimmed final_state from run:complete for single-leaf runs as an optimisation, while Lightning #4531 had separately added storage for final_dataclip_id without rewiring the sync-mode webhook consumer to fall back to it. Each PR was fine on its own; together, sync-mode webhook responses started coming back empty. Fixed in kit #1333 and pin-bumped in Lightning #4561. Lightning's existing tests would have caught it - they just weren't running against the worker that shipped.
This issue is about closing that pre-publish gap.
What we want
One suite of integration tests, in a standalone repo, run three ways:
- test-repo ↔ production - the harness running against pinned versions on both sides.
- kit main ↔ production Lightning - triggered from kit release tags (or main?).
- Lightning main ↔ production CLI/worker - triggered from Lightning release tags (or again, main).
There's a working prototype at OpenFn/project-integration-tests covering project-merge stories. It has a runner abstraction that lets the same test file execute "do operation X" via either the CLI or a Lightning mix task - the load-bearing pattern that makes "tests as contract" work. The merge stories are a narrow start; this epic extends the harness to cover the run-execution path that actually broke.
Why a standalone repo and not a folder in either
- You can pin "suite vN" against arbitrary historical heads of kit and Lightning to prove regression windows. Hard to do cleanly when tests live in one of the trees.
- The contract belongs to neither side, so the tests probably shouldn't either.
- A single set of workflows that both repos call into, instead of two copies that drift.
A small downside/cost is that contributors occasionally hitting a CI failure on a repo they don't have checked out - is real but acceptable, provided the originating PR shows the failure loudly.
Deployment-time variant
The same suite can run against the actually-configured versions before a worker/Lightning pod rolls out, not just @latest. If we parameterise the workflows by image tag / npm version, this becomes a precondition we can run pre-rollout. Stretch goal for this epic.
But more importantly the test suite should be usable by anyone on their local machines allowing for bisecting or specific version targeting.
Scope
-
Harness foundations
- Add worker-process support to the harness - a fixture that boots
@openfn/ws-workerat a chosen ref (npm version, git sha, or local path) against the test's Lightning, waits for it to connect and start claiming, surfaces logs on failure, and tears down cleanly. Existingcli/lightninginvocations stay as they are; this is a new component that runs alongside them so run-execution stories have a worker to dispatch to. - Parameterise workflows by kit ref, Lightning ref, and (stretch) npm/image version so all three run modes share one suite.
- Cross-repo triggers from kit and Lightning release branches.
- Add worker-process support to the harness - a fixture that boots
-
Canary sync test
- One end-to-end story exercising the sync-mode webhook reply path: boot Lightning (via the existing setup), boot a worker against it (via the new fixture), POST to a
webhook_reply: :after_completionworkflow, assert on the response body rather than just the status. - Wired into both repos' release flows.
- Pinning to the kit/Lightning refs from the recent regression should make it fail; pinning to the fix PRs should make it pass. That's the the 'bisect' test.
- One end-to-end story exercising the sync-mode webhook reply path: boot Lightning (via the existing setup), boot a worker against it (via the new fixture), POST to a
-
Lightning-side dev ergonomics
- Default the Lightning test suite to install worker + CLI at
@latestrather than hard-pinning inassets/package.json/install_runtime.ex. Pair with a lockfile so installs are reproducible. - Lightweight nudge (dev-startup check or git hook) when the installed
@latestdrifts from npm@latest, so the warning is local and contextual rather than CI noise. - Goal: devs working on anything that touches the worker/CLI surface are running a current version by default, without anyone having to remember to bump a pin.
- Default the Lightning test suite to install worker + CLI at
-
Notification
- Failures on the standalone repo need to surface on the originating PR/release. A status icon alone isn't enough.
- Fallback if needed: post-tag job on each repo that refuses to release if the last cross-repo run failed (gh API query).
Not in this epic
- Lightning #4559 - decoupling
handle_delayed_responsefrom worker-suppliedfinalState. Removes this regression class entirely; complementary, not a substitute. Tracked there.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.