graphile / graphile/graphile-engine
Making live-queries (ie. subscription-lds) more efficient/scalable
- Dominant language
- JavaScript
- Stars
- 764
- Forks
- 128
- PR merge metrics
- No merged PRs in 30d
Description
### Feature description
The live-query feature is impressive, but it appears to be lacking a number of optimizations needed for high scalability.
I know that most functionality that's achieved through live-queries can be approximated through use of the more efficient TRIGGER/NOTIFY subscriptions; however, the live-query approach is much more "elegant" in my view, and it requires less "wiring" to set up each time.
Because live-queries are not a high-priority feature for most developers, I expect that I will need to develop most of the optimizations myself. This will probably be a lot of work, but I'd like to give a try at it, as if I manage it, I can then benefit from the optimizations in all my future projects. (as well as the ~half dozen projects I currently have on Firestore that I plan to migrate)
This thread is my attempt to list those potential optimizations, discuss them, and eventually attempt to develop support for them. (either by enhancing the existing subscription-lds plugin, or creating a new plugin that sits on top of it)
### Motivating example
What motivated this feature request for me, is that I am attempting to transition a project from Google Firestore to Postgraphile + Postgres, and its codebase is heavily reliant on the live-query functionality that Firestore supports.
I'm not in a rush to complete the transition (the current site works fine for now), but I would like to gradually extend Postgraphile until its live-query support can scale to a level comparable to Firestore. (granted, it will likely be more expensive and require horizontal-scaling/dedicated-live-query-servers, but that's acceptable)
Like mentioned, most developers seem to get by with the simpler TRIGGER/NOTIFY subscriptions. However, I think if the live-query functionality were optimized enough that you could use it on larger production sites, a lot more developers would be interested in using it.
From what I understand, Firebase/Firestore and Baqend/InvaliDB are the only two database solutions which have [thoroughly
scalable live-queries](https://medium.baqend.com/real-time-databases-explained-why-meteor-rethinkdb-parse-and-firebase-dont-scale-822ff87d2f87) at the moment. And as Firestore and InvaliDB are both proprietary, this makes scalable live-query support in Postgraphile particularly desirable.
The [Medium article above](https://medium.baqend.com/real-time-databases-explained-why-meteor-rethinkdb-parse-and-firebase-dont-scale-822ff87d2f87), written by Wolfram Wingerath (the developer of InvaliDB's live-query solution), explains what some of those necessary optimizations are. (beyond the Medium article, he's also written papers on the subject, which describe the challenges and solutions in greater depth)
### Breaking changes
It depends on the implementation details. The optimizations will likely require some architectural extensions (eg. the concept of "partitioned live-query servers"), but most of these changes should be able to be implemented in external packages, rather than the [subscription-lds](https://github.com/graphile/graphile-engine/tree/v4/packages/subscriptions-lds) and [lds](https://github.com/graphile/graphile-engine/tree/v4/packages/lds) packages themselves. (For now, I'll just maintain an independent fork with the changes needed.)
### Supporting development
I [tick all that apply]:
- [x] am interested in building this feature myself
- [x] am interested in collaborating on building this feature
- [x] am willing to help testing this feature before it's released
- [x] am willing to write a test-driven test suite for this feature (before it exists)
- [ ] am a [Graphile sponsor](https://www.graphile.org/sponsor/) ❤️
- [ ] have an active [support or consultancy contract](https://www.graphile.org/support/) with Graphile
My next post will attempt to describe what some of the desired optimizations are.
Contributor guide
Assessment
This issue has not been assessed yet.