Increasing the citus.max_adaptive_executor_pool_size could perform much worse under high concurrency
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
When `citus.max_adaptive_executor_pool_size = 16`
```
scaling factor: 1
query mode: simple
number of clients: 128
number of threads: 32
duration: 70 s
number of transactions actually processed: 76210
latency average = 117.207 ms
latency stddev = 246.678 ms
tps = 1083.331588 (including connections establishing)
tps = 1084.656088 (excluding connections establishing)
```
When `citus.max_adaptive_executor_pool_size = 1`
```
scaling factor: 1
query mode: simple
number of clients: 128
number of threads: 32
duration: 70 s
number of transactions actually processed: 202165
latency average = 44.135 ms
latency stddev = 32.803 ms
tps = 2884.417599 (including connections establishing)
tps = 2887.712440 (excluding connections establishing)
```
Schemas & scripts to run:
```SQL
e steps to create the steps I followed
-- tables and load 1M rows
CREATE TABLE table_1 (key int primary key, value int);
SELECT create_distributed_table('table_1', 'key');
INSERT INTO table_1 SELECT i, i % 1000 FROM generate_series(0, 1000000)i;
-- create index on the table that we'll use as an index
CREATE INDEX table_1_value ON table_1 (value);
-- multi shard lookup queries on the index
cat mshard_lookup.sql
\set aid random(0, 1000000)
SELECT count(*) FROM table_1 WHERE value = :bid;
-- lets run lots of simple lookups, covering all the multi-shard join times
pgbench -f mshard_lookup.sql -c 128 -j 32 -T 150 -P 1 "xXXX"
```
Contributor guide
Assessment
This issue has not been assessed yet.