graphile / graphile/worker

Race in add_job/add_jobs with keyed jobs can drop scheduling requests (and returns no row under contention)

Open
#580 3 comments 0 reactions 0 assignees View on GitHub
🐛 bug
Dominant language
TypeScript
Stars
2.4k
Forks
126
Avg merge
2d 2h
Merged PRs (30d)
9

Description

### Summary

I am the maintainer of [graphile_worker_rs](https://github.com/leo91000/graphile_worker_rs), a Rust rewrite of `graphile/worker`.

A user recently reported a bug, and I think this exposes a race condition in the keyed scheduling path (`add_jobs` / `add_job`).

Original report: https://github.com/leo91000/graphile_worker_rs/issues/378

### Steps to reproduce

1. Start PostgreSQL and initialize Graphile Worker schema.
2. Run a worker with concurrency `10` and two tasks:
- `printer`: very short task (e.g. sleeps ~2ms) so keyed jobs frequently become locked/running.
- `scheduler`: loops many times (e.g. `100`), each time calling `addJob("printer", { key }, { jobKey: key, jobKeyMode: "preserve_run_at" })` with key chosen from a small keyspace (e.g. 10 keys).
3. Enqueue multiple `scheduler` jobs concurrently (e.g. 4).
4. Let it run for ~30-60 seconds.

This creates high contention on the same `jobKey` while some conflicting rows are locked.

### Expected results

- `addJob(...)` should always return a valid job row.
- For `replace` / `preserve_run_at`, scheduling should not occasionally return “no row” under contention.

### Actual results

Under contention, `graphile_worker.add_jobs(...)` can return no row for a spec because of:

- `ON CONFLICT (key) DO UPDATE ... WHERE jobs.locked_at IS NULL`

When that `WHERE` condition is false, the conflict path does nothing and returns nothing for that spec.
Then `add_job(...)` (which selects from `add_jobs(...) LIMIT 1`) can return a null/empty row.

In strict clients this surfaces clearly (example from Rust/sqlx):
- `error occurred while decoding column "id": unexpected null; try decoding as an Option`

In JS this can manifest as `rows[0]` missing from `add_job(...)` result in edge cases.

### Additional context

- Reproduced from issue `https://github.com/leo91000/graphile_worker_rs/issues/378`
- Reproduced against upstream SQL shape in `sql/000018.sql`:
- `add_job` delegates to `add_jobs` (`select * into v_job from ...add_jobs(...)`)
- `add_jobs` uses `ON CONFLICT (key) DO UPDATE ... WHERE jobs.locked_at is null`
- Current repo version checked: `0.17.0-rc.0` (from `package.json`)
- PostgreSQL: reproduced on Docker Postgres (15/16 class behavior)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.