citusdata / citusdata/citus

Pipeline INSERT (UPDATE / UPSERT) statements

Open
#369 0 comments 0 reactions 1 assignee Claimed by @anarazel View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

When the user issues an `INSERT`, we route the `INSERT` to all worker nodes that have the shard in the same order. We then wait for both replicas to acknowledge the write and return to the user. This approach has notable latency and throughput implications.

We have several ideas around pipelining `INSERT` statements:
1. The client already has pipelining support. For example, the JDBC client uses PostgreSQL's pipelining protocol to prepare, bind, execute, bind, execute, and then wait for the results.
2. The master node implements pipelining. This works if we have many concurrent requests coming into the system -- we can then create a per shard queue, batch inserts and then flush them. Once we flush all `INSERT`s, we acknowledge to the user.
3. We store the data on the master node in a queue table. This table keeps each statement's logical representation. We then flush the data at regular intervals. This approach works as long as we have a single master node. We also need to figure out if we'd like to return stale results on `SELECT`s.
4. Do we wait for an ACK? Do we want to introduce logical decoding?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.