citusdata / citusdata/citus

Consider making multi-row INSERT parallel and concurrent

Open
#3,315 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

We've previously decided to run multi-row inserts across shards sequentially to allow high concurrency and enable high throughput. Running the inserts in parallel had high overhead due to the large number of connections to the worker, and the aggressive locking to avoid deadlocks. However, sequential execution performs poorly for large multi-row inserts.

With the adaptive executor, we might reconsider our position. We can run multi-row inserts with limited parallelism using one connection per worker. In practice, multi-row inserts (without ON CONFLICT) are not very likely to create deadlocks since that would typically only happen if they are e.g. inserting the same values into a column with a unique constraint, which will always fail.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.