step:complete crashes with Postgrex 22P05 when output_dataclip body contains a NUL byte
Nobody has claimed this yet.
- Dominant language
- Elixir
- Stars
- 296
- Forks
- 86
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 50
Description
Describe the bug
When a worker reports step:complete with an output_dataclip whose JSON contains a NUL byte (\u0000), the dataclip insert crashes with an unhandled Postgrex.Error and the step never completes:
Postgrex.Error: ERROR 22P05 (untranslatable_character)
unsupported Unicode escape sequence
\u0000 cannot be converted to text.
Sentry: LIGHTNING-13C (48 events, first seen 2025-08-07, still occurring as of 2026-06-24; prod, app.openfn.org).
Version number v2.16.6 (1.5.86) (from the Sentry event); reproduced on current main.
I have reproduced this locally on main:
- Yes
- No
Root cause
The crash is in Lightning.Runs.Handlers.CompleteStep, inserting the Dataclip body (lib/lightning/runs/handlers.ex:467-482):
Dataclip.new(%{
id: dataclip_id,
project_id: project_id,
body: output_dataclip |> Jason.decode!() |> ensure_map(), # <- NUL byte ends up here
type: :step_result
})
|> Repo.insert()
Stacktrace (confirmed via Sentry):
CompleteStep.call/2:398
-> update_step/2:410 (maybe_save_dataclip)
-> Ecto.Repo.Schema.do_insert/4
-> Ecto.Adapters.SQL.raise_sql_call_error/1
The worker sends literal JSON text containing the 6-character escape \u0000 (valid JSON — this is not a JSON-validity problem). Jason.decode! turns that escape into a real NUL byte in the Elixir binary. PostgreSQL jsonb/text physically cannot store a NUL byte (0x00), so the insert raises. Because the error is unhandled, the whole step:complete reply fails and the step/run is left without a clean completion.
To Reproduce
A step:complete payload of the form (note output_dataclip carries the escape):
{
"step_id": "...",
"run_id": "...",
"project_id": "...",
"reason": "success",
"output_dataclip_id": "...",
"output_dataclip": "{\"value\": \"foo\\u0000bar\"}"
}
A reproduction test (currently asserts the crash):
test "reproduces LIGHTNING-13C: output_dataclip with a NUL byte crashes the insert" do
# ... set up workflow/run/step ...
output_dataclip = ~s({"value": "foo\\u0000bar"})
assert Jason.decode!(output_dataclip) == %{"value" => "foo\0bar"}
assert_raise Postgrex.Error, ~r/unsupported Unicode escape sequence/, fn ->
Runs.complete_step(%{
step_id: step.id, reason: "success",
output_dataclip: output_dataclip,
output_dataclip_id: Ecto.UUID.generate(),
run_id: run.id, project_id: workflow.project_id
})
end
end
This passes — confirming the exact error class, code path, and message match Sentry.
Expected behavior
The step/run should complete successfully without crashing. NUL bytes (and other PostgreSQL-incompatible control chars) in the dataclip body should be handled gracefully rather than raising an unhandled error.
Proposed handling
There's already a precedent: Lightning.LogMessage (lib/lightning/ecto_types.ex:63) sanitizes exactly this for the run:log path — stripping NUL bytes, control chars, and literal \u0000 escapes, replacing them with the Unicode replacement character. The dataclip body never got the same protection.
Options:
- A — targeted: sanitize the raw
output_dataclipstring beforeJason.decode!inmaybe_save_dataclip, reusing theLogMessageregex logic. - B — boundary (recommended): a custom jsonb Ecto type (the map analogue of
LogMessage) that recursively sanitizes string keys/values, applied toDataclip.body. This protects all dataclip write paths (webhook-received dataclips, run input dataclips, step results), not justCompleteStep, which can hit the same NUL crash elsewhere.
Open product question: sanitize-and-keep (replace NUL, run completes — consistent with logs) vs reject (return an error reply; avoids silently mutating output but loses the whole output and risks stuck runs). Recommendation: sanitize, matching existing log behaviour.
Additional context
error_messageis commented out of the Step schema (step.ex:55) and not persisted, so it can't trigger this; the dataclipbody(jsonb) is the only crashing path observed.error_type/exit_reasonare short:stringcodes — same class, low risk.- Retained Sentry events cluster suspiciously: 4 within ~12 min on 2026-06-23 and 5 within ~17 min on 2026-04-15, suggesting individual workflows repeatedly emitting NUL-containing output.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in lib/lightning/runs/handlers.ex around lines 467-482 and the CompleteStep path, then inspect the LogMessage sanitization in lib/lightning/ecto_types.ex:63 and the reproduction test described in the issue. Confirm the Dataclip.body write paths and define completion behavior for NUL-containing output, including whether sanitization or rejection is required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir, postgresql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100