payloadcms / payloadcms/payload

Jobs: failed payload-jobs-log write crashes the worker process via handleWorkflowError (uncaught `TypeError: Cannot use 'in' operator to search for '_rels' in undefined`), stranding the job at processing=true

Open
#17,645 0 comments 0 reactions 1 assignee View on GitHub

@AlessioGr is already working on this.

Since Aug 5, 2026.

Dominant language
TypeScript
Stars
44.8k
Forks
4.2k
Avg merge
2d 21h
Merged PRs (30d)
53

Description

Describe the Bug

When a running job's payload_jobs_log append fails at the database level, the error-handling path itself crashes the whole worker process instead of failing the job:

  1. The jobs-log insert fails (in our case: a unique (_parent_id, _order) collision — see trigger context below) → WorkflowError.
  2. handleWorkflowErrorupdateJobs then throws TypeError: Cannot use 'in' operator to search for '_rels' in undefineduncaught, taking the Node process down.
  3. The crashed worker's claimed job row is left at processing: true forever — there is no lease/expiry/heartbeat mechanism — so with enableConcurrencyControl and an exclusive concurrency key, every future job sharing that key is skipped as "already running" indefinitely. The queue for that key is permanently wedged until someone deletes the row by hand.

Two distinct defects, independent of what causes the log write to fail:

  • An error handler should never itself throw an uncaught exception; a failed log append should fail/mark that job, not kill the process (which may be running other jobs' queues too).
  • A processing: true row has no recovery semantics, so any hard crash (this one, OOM kill, SIGKILL during deploy) permanently wedges concurrency-keyed queues.
Trigger context (not re-reporting the race)

We hit the log-write collision through the non-atomic claim race from #16043 — two workers during a zero-downtime deploy overlap both claimed and ran the same jobs, and the loser's log _order collided with the winner's. We're aware #17441 fixes the claim race itself (thank you!) — as of v3.87.0 that fix is not yet in a published release, and this report is about the failure path, which any jobs-log write failure can trigger (transient DB error, constraint violation, connection drop mid-run), and about the unrecoverable processing: true state any process crash leaves behind.

Observed evidence (v3.87.0, Postgres)

Production-like staging, 2026-08-04 (UTC, instance-labeled logs): deploy overlap; instance A executed 4 jobs 21:14:19–20Z; instance B executed 3 of the same jobs again 21:14:21Z, then logged Error running job Task: buildSite id: 91 (failed jobs-log insert) followed by the _rels TypeError crash banner and process exit. Job 91's row remained processing: true permanently; its exclusive concurrency key blocked every subsequent job for that key.

To Reproduce

On v3.87.0 (any version before the #17441 fix ships), Postgres adapter:

  1. Two Node processes sharing one database, both running the same queue — e.g. two payload jobs:run --cron "* * * * *" processes (top-of-minute ticks make the race near-certain).
  2. Queue several jobs of a task with retries: 0 and an exclusive concurrency key, enableConcurrencyControl: true.
  3. Both processes claim overlapping jobs; the loser's jobs-log insert collides; its process dies with the uncaught _rels TypeError; its claimed rows stay processing: true forever and the concurrency key never frees.

After the #17441 release, step 3's trigger disappears, but the same crash-and-wedge shape is reachable via any mid-run jobs-log write failure or a hard process kill.

Expected behavior
  • A failed jobs-log write marks/fails that job; handleWorkflowError never throws uncaught and never takes the process down.
  • processing: true carries some lease/heartbeat/expiry semantics so a crashed worker's claims are recoverable without manual row deletion.
Environment
  • payload: 3.87.0
  • @payloadcms/db-postgres (Postgres 18)
  • Node 24; two worker processes only during zero-downtime deploy replacement windows
Workarounds we ship (for anyone else hitting this)
  • Drain-aware runner: SIGTERM stops the outgoing instance claiming anything new (shrinks the overlap to seconds).
  • A periodic sweep that deletes payload-jobs rows stuck at processing: true for >15 minutes and closes their downstream records — this is what un-wedges concurrency-keyed queues after a crash.
  • Job side effects made idempotent/immutable, so a duplicate execution wastes work but cannot corrupt state.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.