Implement “delayed/batch insert view”
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
One area that needs particular attention with Citus is DML, where the concurrency semantics of Citus are making multi-shard DML operation hard to scale. In some cases though, users might be able to accept non-transactional or asynchronous semantics in order to obtain a much higher throughput for their INSERT traffic.
The idea we had in a discussion with @marcocitus is to implement a new Citus facility: a VIEW or a TABLE that accepts INSERT traffic locally on the coordinator, as a “buffer”, and then batch-apply the inserts as distributed COPY statements internally from the coordinator to the workers.
The buffer could be a _logical decoding_ facility and the batch-apply process could be implemented as a _background worker_ maybe.
For UPDATE and DELETE traffic the semantics of having a buffer are more complex, in particular the error handling aspects of it. With the INSERT, the Coordinator can run the command to completion on a local PostgreSQL table to ensure timely error reporting --- if it made it there, it should make it to the shards.
Finally, this technique might be also used for _rollup_ use cases, where you can compute rollup for a batch at a time. See #1935.
Contributor guide
Assessment
This issue has not been assessed yet.