Consider making multi-row INSERT parallel and concurrent
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
We've previously decided to run multi-row inserts across shards sequentially to allow high concurrency and enable high throughput. Running the inserts in parallel had high overhead due to the large number of connections to the worker, and the aggressive locking to avoid deadlocks. However, sequential execution performs poorly for large multi-row inserts.
With the adaptive executor, we might reconsider our position. We can run multi-row inserts with limited parallelism using one connection per worker. In practice, multi-row inserts (without ON CONFLICT) are not very likely to create deadlocks since that would typically only happen if they are e.g. inserting the same values into a column with a unique constraint, which will always fail.
Contributor guide
Assessment
This issue has not been assessed yet.