step_created entity row and event are written non-transactionally — a crash between them wedges the run permanently

step_created entity row and event are written non-transactionally — a crash between them wedges the run permanently

Summary

In @workflow/world-postgres, events.create() writes a step as two separate statements with no transaction:

1. workflow_steps — the entity row (the exactly-once ownership claim)
2. workflow_events — the step_created row (replay's source of truth)

A crash between them leaves an entity row with no step_created event. That state is unrecoverable: the run stays running with a pending step and replays forever without progressing.

You already recognise and repair this exact partial write on the hook_created path — the step_created path has no equivalent branch.

Why it deadlocks

The two sources of truth disagree, and each side is behaving correctly:

• Replay rebuilds from the event log. No step_created → it concludes the step was never created and re-issues it as a lazy step_started (one carrying stepName + input) — @workflow/core dist/runtime/step-executor.js:255.
• The lazy-start exactly-once gate reads the entity row, finds the orphan, concludes a concurrent worker won the create, and throws — dist/storage.js:526:

if (lazyStepStart && validatedStep) {
  throw new EntityConflictError(`Step "${data.correlationId}" already created`);
}

• step-executor.js:152-163 maps EntityConflictError to { type: 'skipped' }, and the aggregate loop re-replays with no backoff.

So the log says "never created, run it", the row says "already claimed, skip it", and every replay re-derives the same standoff. There is no path out — no retry, no timeout, no reaper in the library, and the run never reaches a terminal state.

Your own precedent

dist/storage.js:1042, on the hook path:

Idempotency: if the existing hook is the same (runId, hookId) we are trying to create, this is either a duplicate / replayed processing of the same hook_created (not a real conflict), or an orphaned hook row from a prior crashed attempt (the hook INSERT below landed but the events INSERT below didn't — these writes are not in one transaction). Distinguish by checking whether the hook_created event actually exists in the event log … missing → orphaned hook row (crash between hook INSERT and events INSERT): skip the hook insert … and fall through to the events INSERT below, completing the partial write.

That's the correct handling. step_created (storage.js:811) and the lazy-start claim (storage.js:857) both use bare onConflictDoNothing() and never check whether the creation event exists.

Production evidence

Self-hosted world-postgres, ~12M journal inserts/day.

• 15 step entity rows with no step_created event accumulated over 11 days (~1.4/day). Same DB: 0 orphaned hooks (your fix works) and 0 orphaned waits.
• One instance wedged a live customer session for 7 hours. The workflow body re-executed ~500×/sec the entire time, pegging a worker thread. Only manual DB intervention cleared it.
• On the wedged run, the step had a workflow_steps row and the event log simply stopped at the previous step's step_completed — no step_created for it, ever.

The window is not small in practice. From pg_stat_statements on that database:

statement	mean	max
insert into workflow_events	12–364 ms	94.6 s
insert into workflow_steps	87 ms	129.1 s

Under load the gap between the two inserts can exceed a minute, so any process termination in that window (deploy, OOM, SIGKILL, connection loss) can produce this.

Suggested fix

Either would work:

1. Make the two writes atomic — wrap the entity insert and its creation event in one transaction. This is the real fix and would cover step_created, wait_created, and attr_set, which all share the shape.
2. Port the hook branch to steps — on a zero-row onConflictDoNothing(), check whether the step_created event exists. Present → genuine concurrent create, throw EntityConflictError as today. Missing → orphaned partial write, complete it instead of throwing.

Note wait_created and attr_set are written the same way and are covered by the same partial index (workflow_events_entity_creation_unique), so (1) is the durable answer.

Workaround we're running

We wrap world.events in our composed world (no patch to your package) and, on that specific conflict, re-issue the missing step_created through your own create() using the payload the intercepted lazy step_started already carries. The original error is still rethrown — the gate keys off the entity row, so that call can't proceed — and the next replay sees the event, emits a plain step_started, and runs the step.

We deliberately did not delete the orphan row: proving a row is abandoned requires an age threshold larger than the max insert latency above, which means minutes of user-visible stall, and losing that race inverts the partial write into an event with no row.

Environment

• @workflow/world-postgres 5.0.0-beta.20
• @workflow/core 5.0.0-beta.25
• Self-hosted Postgres world, graphile-worker queue, Node 22