OpenFn / OpenFn/lightning

step:complete crashes with Postgrex 22P05 when output_dataclip body contains a NUL byte

Open
#4,893 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Community Contribution 🏅
Dominant language
Elixir
Stars
296
Forks
86
Avg merge
1d 13h
Merged PRs (30d)
50

Description

Describe the bug

When a worker reports step:complete with an output_dataclip whose JSON contains a NUL byte (\u0000), the dataclip insert crashes with an unhandled Postgrex.Error and the step never completes:

Postgrex.Error: ERROR 22P05 (untranslatable_character)
unsupported Unicode escape sequence
\u0000 cannot be converted to text.

Sentry: LIGHTNING-13C (48 events, first seen 2025-08-07, still occurring as of 2026-06-24; prod, app.openfn.org).

Version number v2.16.6 (1.5.86) (from the Sentry event); reproduced on current main.

I have reproduced this locally on main:

  • Yes
  • No

Root cause

The crash is in Lightning.Runs.Handlers.CompleteStep, inserting the Dataclip body (lib/lightning/runs/handlers.ex:467-482):

Dataclip.new(%{
  id: dataclip_id,
  project_id: project_id,
  body: output_dataclip |> Jason.decode!() |> ensure_map(),  # <- NUL byte ends up here
  type: :step_result
})
|> Repo.insert()

Stacktrace (confirmed via Sentry):

CompleteStep.call/2:398
  -> update_step/2:410          (maybe_save_dataclip)
    -> Ecto.Repo.Schema.do_insert/4
      -> Ecto.Adapters.SQL.raise_sql_call_error/1

The worker sends literal JSON text containing the 6-character escape \u0000 (valid JSON — this is not a JSON-validity problem). Jason.decode! turns that escape into a real NUL byte in the Elixir binary. PostgreSQL jsonb/text physically cannot store a NUL byte (0x00), so the insert raises. Because the error is unhandled, the whole step:complete reply fails and the step/run is left without a clean completion.

To Reproduce

A step:complete payload of the form (note output_dataclip carries the escape):

{
  "step_id": "...",
  "run_id": "...",
  "project_id": "...",
  "reason": "success",
  "output_dataclip_id": "...",
  "output_dataclip": "{\"value\": \"foo\\u0000bar\"}"
}

A reproduction test (currently asserts the crash):

test "reproduces LIGHTNING-13C: output_dataclip with a NUL byte crashes the insert" do
  # ... set up workflow/run/step ...
  output_dataclip = ~s({"value": "foo\\u0000bar"})
  assert Jason.decode!(output_dataclip) == %{"value" => "foo\0bar"}

  assert_raise Postgrex.Error, ~r/unsupported Unicode escape sequence/, fn ->
    Runs.complete_step(%{
      step_id: step.id, reason: "success",
      output_dataclip: output_dataclip,
      output_dataclip_id: Ecto.UUID.generate(),
      run_id: run.id, project_id: workflow.project_id
    })
  end
end

This passes — confirming the exact error class, code path, and message match Sentry.

Expected behavior

The step/run should complete successfully without crashing. NUL bytes (and other PostgreSQL-incompatible control chars) in the dataclip body should be handled gracefully rather than raising an unhandled error.

Proposed handling

There's already a precedent: Lightning.LogMessage (lib/lightning/ecto_types.ex:63) sanitizes exactly this for the run:log path — stripping NUL bytes, control chars, and literal \u0000 escapes, replacing them with the Unicode replacement character. The dataclip body never got the same protection.

Options:

  • A — targeted: sanitize the raw output_dataclip string before Jason.decode! in maybe_save_dataclip, reusing the LogMessage regex logic.
  • B — boundary (recommended): a custom jsonb Ecto type (the map analogue of LogMessage) that recursively sanitizes string keys/values, applied to Dataclip.body. This protects all dataclip write paths (webhook-received dataclips, run input dataclips, step results), not just CompleteStep, which can hit the same NUL crash elsewhere.

Open product question: sanitize-and-keep (replace NUL, run completes — consistent with logs) vs reject (return an error reply; avoids silently mutating output but loses the whole output and risks stuck runs). Recommendation: sanitize, matching existing log behaviour.

Additional context

  • error_message is commented out of the Step schema (step.ex:55) and not persisted, so it can't trigger this; the dataclip body (jsonb) is the only crashing path observed. error_type/exit_reason are short :string codes — same class, low risk.
  • Retained Sentry events cluster suspiciously: 4 within ~12 min on 2026-06-23 and 5 within ~17 min on 2026-04-15, suggesting individual workflows repeatedly emitting NUL-containing output.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lib/lightning/runs/handlers.ex around lines 467-482 and the CompleteStep path, then inspect the LogMessage sanitization in lib/lightning/ecto_types.ex:63 and the reproduction test described in the issue. Confirm the Dataclip.body write paths and define completion behavior for NUL-containing output, including whether sanitization or rejection is required.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir, postgresql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.