Remove `:global` dataclip type
Nobody has claimed this yet.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
User story
As a developer maintaining Lightning, I want a single, coherent model for dataclips so the wipe and retention pipelines don't need type-specific carve-outs that produce latent bugs like #4795.
Details
The :global dataclip type exists alongside :http_request, :step_result, :saved_input, and :kafka, but its actual semantics are narrow: a system-created, unnamed dataclip that should never be wiped per-run. Its only producer is Lightning.Workflows.Scheduler.invoke_cronjob/1, which seeds the first cron run with %{type: :global, body: %{}}.
That partial support is the root of #4795: Lightning.Invocation.Query.wipe_dataclips/1 excludes :global from its type allowlist, so on :erase_all projects the per-run wipe affects zero rows and Runs.wipe_dataclips/1's {1, [dataclip]} pattern crashes. #4796 patched the crash with a {0, _} no-op clause, but the underlying modeling tension remains: :global is just "a system-created, unnamed saved input that we never wipe per-run."
Implementation notes
Removing :global entirely is the goal. A suggested implementation (open to alternatives):
- Add
belongs_to :created_by, UsertoDataclip, mirroringRun.created_by(nullable column). A blankcreated_byin the UI means "created by the system" — same convention used for runs today. - Change
Scheduler.invoke_cronjob/1to seed:saved_inputwithcreated_by: nilinstead of:global. - Drop
:globalfromDataclip.@source_types. Data-migrate existing rows to:saved_input(leavingcreated_by_idNULL). - Simplify
Lightning.Invocation.Query.wipe_dataclips/1so the only exclusion isis_nil(d.name)— named saved inputs never get wiped, everything else does.
Callsites that will need changes:
lib/lightning/invocation/dataclip.ex— schema +@source_types+ docstringlib/lightning/workflows/scheduler.ex:90-97— cron-first-run seedlib/lightning/invocation/query.ex:316-324—wipe_dataclips/1lib/lightning/runs.ex:182— keep the{0, _}no-op clause (named dataclips can still produce zero-row updates)lib/lightning/projects/sandboxes.ex:97—@allowed_dataclip_typeslib/lightning_web/live/components/common.ex:574andlib/lightning_web/live/components/icon.ex:18— display enum and badge styling- Tests under
test/lightning/workflows/scheduler_test.exs,test/lightning/sandboxes_test.exs,test/lightning/invocation_test.exs,test/lightning_web/controllers/dataclip_controller_test.exs,test/lightning_web/channels/run_channel_test.exs
Open questions
on_deletefordataclips.created_by_id::restrict(matches Run, see migration20260319103734) or:nilify_all?- Should
:kafkadataclips be wiped by the simplified query? They are excluded today under the explicit type allowlist; with anis_nil(d.name)-only filter they would be included.
Release notes
Internal cleanup. The :global dataclip type is removed; existing global dataclips are migrated to :saved_input and remain accessible to jobs and sandboxes as before. Adds a created_by association on dataclips (nullable) so the UI can later distinguish user-created from system-seeded saved inputs.
User acceptance criteria
-
:globalno longer appears inLightning.Invocation.Dataclip.@source_types. - Existing
:globalrows have been migrated to:saved_inputwithcreated_by_idNULL. - The first cron run of a previously-unrun cron workflow seeds a
:saved_inputdataclip withcreated_by_idNULL. -
fetch:dataclipon an:erase_allproject with any non-named dataclip succeeds and wipes the body. - Named dataclips (saved inputs with a
name) are never wiped byRuns.wipe_dataclips/1.
Related
- #4795 — the crash that surfaced this modeling tension
- #4796 — the
{0, _}patch
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with lib/lightning/invocation/dataclip.ex and trace the scheduler, wipe query, sandbox, and LiveView callsites listed in the issue. Run the specified scheduler, sandbox, invocation, controller, and channel tests; the work is done when global rows and cron seeds use saved_input, named dataclips remain protected, and erase_all wipes eligible dataclips without crashing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir
- Domain
- backend, database, frontend
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100