glenjamin / glenjamin/postgres-skip-locked-surprise
Why this happens, and why the INNER JOIN workaround fixes it
- Dominant language
- Go
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Short version: this isn't a `SKIP LOCKED` bug. The lock is on `accounts`, but the
predicate deciding eligibility lives in `imports`, and a row lock only protects
the row it's on.
## The original query
Two workers, account 5, no import row yet.
While A holds the lock on the `accounts` row, B's `LockRows` node can't lock it,
so `SKIP LOCKED` skips. This is the case that gets tested, and it works.
Once A commits, the lock is gone. B reaches `LockRows`, finds the row free, and
takes it. The `accounts` tuple was never modified — A wrote to `imports` — so no
`EvalPlanQual` recheck fires. The join condition over `imports` was evaluated
once, against B's statement snapshot taken before A committed, and nothing
revisits it. B matches account 5 as well.
Even if the `accounts` row had been updated, this would still happen: EPQ
re-evaluates the qual against the updated tuple of the **locked** relation. An
unlocked side of a join is still read from the original snapshot.
## Why the workaround works
Adding `imports` to the lock is the part doing the work. Now the row carrying the
predicate is a locked relation, so both mechanisms finally apply to it: if A
still holds it, `SKIP LOCKED` skips; if A has committed, the row was updated, EPQ
fires, the qual is re-evaluated against the new version, `pending` fails the
`completed` test, and the row drops out.
But that only works once the LEFT JOIN is gone, and that isn't incidental. **An
account with no import row has nothing to lock.** `FOR UPDATE OF imports` cannot
lock, skip, or recheck a tuple that does not exist. A predicate about absence
can't be protected by row locks at all, because absence isn't a row — which is
why dropping that case was necessary before the rest of the fix could hold.
So the rule is: a row lock protects a predicate only if the predicate lives on a
row you locked.
## Keeping the "no import yet" case
Materialize it. Create the `imports` row up front in a `none`/`ready` state
instead of leaving it absent, so eligibility becomes a value on a row rather than
the lack of one. Then `INNER JOIN` + lock both tables covers every account, and
there's no special case left.
Where a predicate is *inherently* about other rows — "no sibling import for this
account is already in flight" — no lock placement helps, because there's no
single row to put it on. The workable pattern there is to claim optimistically,
then re-check in a second statement on a fresh snapshot and back out if you lost.
Two racers may both back out and waste a round, but they can't both proceed,
since at least one re-check sees the other's committed claim.
## One trap when locking a parent row
If you serialize on `accounts` rather than `imports`, watch the lock strength.
Every insert into `imports` takes a `KEY SHARE` lock on its parent via the
foreign key, and `FOR UPDATE` conflicts with `KEY SHARE` — so `FOR UPDATE` on
`accounts` will stall inserts, and anyone holding a write transaction open blocks
all claims. `FOR NO KEY UPDATE` self-conflicts, which is all the exclusivity you
need, and doesn't fight the FK.
---
I hit the same shape in a Postgres-backed webhook queue: parent locked, predicate
in the child, 195 double-claims in 10k across two workers. Zero once the lock
moved onto the row carrying the predicate, with the cross-row guard handled by
the re-verify above. Worked example:
https://github.com/sankalp771/harkara/blob/main/src/worker.ts
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names no repository files or tests; begin with the original query and locking scenario described in the body, then inspect the linked src/worker.ts example. Confirm the PostgreSQL behavior and workaround, but the intended documentation or code deliverable is not specified, so completion needs maintainer clarification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql
- Domain
- databases
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100