Suggestion: Use WAL instead of LISTEN/NOTIFY
- Dominant language
- TypeScript
- Stars
- 2.4k
- Forks
- 126
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 9
Description
### Feature description
This is a big ask, but `LISTEN/NOTIFY` can become a bottleneck for Graphile Worker because it acquires a global lock.
- https://www.recall.ai/blog/postgres-listen-notify-does-not-scale
- https://x.com/samokhvalov/status/1959468091035009245
Maybe it could be replaced by subscribing to the WAL stream of the jobs table? This could increase the rate at which jobs are inserted. Especially with local queues, INSERT could become the new bottleneck.
### Motivating example
Not seeing issues myself yet, but better be safe than sorry.
### Breaking changes
not on the API layer, but this requires a significant rework internally.
### Supporting development
I [tick all that apply]:
- [ ] am interested in building this feature myself
- [x] am interested in collaborating on building this feature
- [x] am willing to help testing this feature before it's released
- [ ] am willing to write a test-driven test suite for this feature (before it exists)
- [x] am a [Graphile sponsor](https://www.graphile.org/sponsor/) ❤️
- [ ] have an active [support or consultancy contract](https://www.graphile.org/support/) with Graphile
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.