citusdata / citusdata/citus

Implement “delayed/batch insert view”

Open
#1,956 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

One area that needs particular attention with Citus is DML, where the concurrency semantics of Citus are making multi-shard DML operation hard to scale. In some cases though, users might be able to accept non-transactional or asynchronous semantics in order to obtain a much higher throughput for their INSERT traffic.

The idea we had in a discussion with @marcocitus is to implement a new Citus facility: a VIEW or a TABLE that accepts INSERT traffic locally on the coordinator, as a “buffer”, and then batch-apply the inserts as distributed COPY statements internally from the coordinator to the workers.

The buffer could be a _logical decoding_ facility and the batch-apply process could be implemented as a _background worker_ maybe.

For UPDATE and DELETE traffic the semantics of having a buffer are more complex, in particular the error handling aspects of it. With the INSERT, the Coordinator can run the command to completion on a local PostgreSQL table to ensure timely error reporting --- if it made it there, it should make it to the shards.

Finally, this technique might be also used for _rollup_ use cases, where you can compute rollup for a batch at a time. See #1935.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.