Wiped dataclips remain searchable — search_vector not cleared on wipe
Nobody has claimed this yet.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
Describe the bug
When a dataclip is wiped (per-run wipe with save_dataclips: false, or the project data-retention job), its body/request are cleared to NULL and wiped_at is stamped, but its full-text search_vector is left intact. The work-order body search matches that stale vector with no wiped_at filter, so a wiped dataclip stays searchable by the exact content that was meant to be erased — a data-retention hole.
Version number
Reproduced on main (commit 6331486e8d) and on the defer-dataclip-search-vector branch (8fc24bab42).
I have reproduced this locally on main:
- Yes
- No
On main the AFTER INSERT trigger builds the vector synchronously; the wipe leaves it stale. On the deferred-indexing branch the worker builds it instead, but the wipe is the same — and the worker only re-sweeps WHERE search_vector IS NULL, so it never re-indexes a wiped row.
To Reproduce
- Create a work order whose input dataclip body contains a distinctive token, e.g.
{"secret": "zzzuniquetoken …"}. - In the work-order list, search that project for
zzzuniquetokenwith the body filter → the work order is found (expected). - Wipe the dataclip — either let the project's data-retention period elapse, or trigger a run with
save_dataclips: false. The dataclip'sbodybecomesNULLandwiped_atis set. - Search again for
zzzuniquetoken(body filter). - The wiped work order is still returned, even though its content is gone.
Expected behavior
Once a dataclip is wiped, searching for terms that were in its (now-erased) body should return nothing — the indexed content should be erased alongside the body.
Screenshots
N/A (confirmed via automated test rather than UI).
Additional context
- Code paths:
- Wipe sets
body/request/wiped_atonly —Invocation.Query.wipe_dataclips/1(lib/lightning/invocation/query.ex), used byRuns.wipe_dataclips/1(lib/lightning/runs.ex) andProjects.wipe_dataclips_for/1(lib/lightning/projects.ex). - Read side has no
wiped_atfilter —Invocation.build_search_fields_where/2,:bodybranch (lib/lightning/invocation.ex), reached from the main UI searchInvocation.search_workorders/3.
- Wipe sets
- Pre-existing, not a regression: the original DB trigger was
AFTER INSERTonly (priv/repo/migrations/20240329123804_…), so it never re-ran on the wipeUPDATE. - A guard already exists, just not wired in:
Invocation.exclude_wiped_dataclips/1filtersis_nil(d.wiped_at), but is only applied tosearch_workorders_for_retry/2and cancel — not the mainsearch_workorders/3. - Fix options: (1) add
search_vector: nilto the wipeupdate_all(on the deferred branch the worker then re-indexes a NULL body to an empty vector → matches nothing) — fixes it at source for every reader; or (2) apply thewiped_atfilter tosearch_workorders/3. Recommend (1), optionally + (2) for defence in depth. Low risk either way.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Invocation.Query.wipe_dataclips/1 in lib/lightning/invocation/query.ex, then read the related paths in lib/lightning/runs.ex, lib/lightning/projects.ex, and lib/lightning/invocation.ex. Run the relevant automated tests and verify that after wiping a dataclip, searching the body for its former distinctive content returns no work order while ordinary body search still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100