graphile / graphile/worker

Suggestion: Use WAL instead of LISTEN/NOTIFY

Open
#559 1 comment 3 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
2.4k
Forks
126
Avg merge
2d 2h
Merged PRs (30d)
9

Description

### Feature description

This is a big ask, but `LISTEN/NOTIFY` can become a bottleneck for Graphile Worker because it acquires a global lock.
- https://www.recall.ai/blog/postgres-listen-notify-does-not-scale
- https://x.com/samokhvalov/status/1959468091035009245

Maybe it could be replaced by subscribing to the WAL stream of the jobs table? This could increase the rate at which jobs are inserted. Especially with local queues, INSERT could become the new bottleneck.

### Motivating example

Not seeing issues myself yet, but better be safe than sorry.

### Breaking changes

not on the API layer, but this requires a significant rework internally.

### Supporting development

I [tick all that apply]:

- [ ] am interested in building this feature myself
- [x] am interested in collaborating on building this feature
- [x] am willing to help testing this feature before it's released
- [ ] am willing to write a test-driven test suite for this feature (before it exists)
- [x] am a [Graphile sponsor](https://www.graphile.org/sponsor/) ❤️
- [ ] have an active [support or consultancy contract](https://www.graphile.org/support/) with Graphile

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.