OpenFn / OpenFn/lightning

Remove `:global` dataclip type

Open
#4,799 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Monitoring Sentry
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

User story

As a developer maintaining Lightning, I want a single, coherent model for dataclips so the wipe and retention pipelines don't need type-specific carve-outs that produce latent bugs like #4795.

Details

The :global dataclip type exists alongside :http_request, :step_result, :saved_input, and :kafka, but its actual semantics are narrow: a system-created, unnamed dataclip that should never be wiped per-run. Its only producer is Lightning.Workflows.Scheduler.invoke_cronjob/1, which seeds the first cron run with %{type: :global, body: %{}}.

That partial support is the root of #4795: Lightning.Invocation.Query.wipe_dataclips/1 excludes :global from its type allowlist, so on :erase_all projects the per-run wipe affects zero rows and Runs.wipe_dataclips/1's {1, [dataclip]} pattern crashes. #4796 patched the crash with a {0, _} no-op clause, but the underlying modeling tension remains: :global is just "a system-created, unnamed saved input that we never wipe per-run."

Implementation notes

Removing :global entirely is the goal. A suggested implementation (open to alternatives):

  • Add belongs_to :created_by, User to Dataclip, mirroring Run.created_by (nullable column). A blank created_by in the UI means "created by the system" — same convention used for runs today.
  • Change Scheduler.invoke_cronjob/1 to seed :saved_input with created_by: nil instead of :global.
  • Drop :global from Dataclip.@source_types. Data-migrate existing rows to :saved_input (leaving created_by_id NULL).
  • Simplify Lightning.Invocation.Query.wipe_dataclips/1 so the only exclusion is is_nil(d.name) — named saved inputs never get wiped, everything else does.

Callsites that will need changes:

  • lib/lightning/invocation/dataclip.ex — schema + @source_types + docstring
  • lib/lightning/workflows/scheduler.ex:90-97 — cron-first-run seed
  • lib/lightning/invocation/query.ex:316-324wipe_dataclips/1
  • lib/lightning/runs.ex:182 — keep the {0, _} no-op clause (named dataclips can still produce zero-row updates)
  • lib/lightning/projects/sandboxes.ex:97@allowed_dataclip_types
  • lib/lightning_web/live/components/common.ex:574 and lib/lightning_web/live/components/icon.ex:18 — display enum and badge styling
  • Tests under test/lightning/workflows/scheduler_test.exs, test/lightning/sandboxes_test.exs, test/lightning/invocation_test.exs, test/lightning_web/controllers/dataclip_controller_test.exs, test/lightning_web/channels/run_channel_test.exs
Open questions
  • on_delete for dataclips.created_by_id: :restrict (matches Run, see migration 20260319103734) or :nilify_all?
  • Should :kafka dataclips be wiped by the simplified query? They are excluded today under the explicit type allowlist; with an is_nil(d.name)-only filter they would be included.
Release notes

Internal cleanup. The :global dataclip type is removed; existing global dataclips are migrated to :saved_input and remain accessible to jobs and sandboxes as before. Adds a created_by association on dataclips (nullable) so the UI can later distinguish user-created from system-seeded saved inputs.

User acceptance criteria
  • :global no longer appears in Lightning.Invocation.Dataclip.@source_types.
  • Existing :global rows have been migrated to :saved_input with created_by_id NULL.
  • The first cron run of a previously-unrun cron workflow seeds a :saved_input dataclip with created_by_id NULL.
  • fetch:dataclip on an :erase_all project with any non-named dataclip succeeds and wipes the body.
  • Named dataclips (saved inputs with a name) are never wiped by Runs.wipe_dataclips/1.
Related
  • #4795 — the crash that surfaced this modeling tension
  • #4796 — the {0, _} patch

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with lib/lightning/invocation/dataclip.ex and trace the scheduler, wipe query, sandbox, and LiveView callsites listed in the issue. Run the specified scheduler, sandbox, invocation, controller, and channel tests; the work is done when global rows and cron seeds use saved_input, named dataclips remain protected, and erase_all wipes eligible dataclips without crashing.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir
Domain
backend, database, frontend
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.