Multi-shard range search query performance issue
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
We observe multi-shard range search query performance issue in our dev citus cluster.
The cluster has one coorindator and 3 workers, all configured with 8 CPUs and 30G memory. To demonstrate the performance issue, we created a simple table as follows (note the table is set to have 16 shards)
```sql
create table person (id bigint not null, name text not null);
select create_distributed_table('person', 'id');
create index on person (id);
```
To test the performance, we create a REST server with a simple endpoint to get the first 100 persons from the database, essentially every request runs the query `select * from person order by id limit 100`.
The number of requests handled per second measured is about 2000/s. Metrics show the coordinator is kept busy with 800% cpu usage and the worker nodes cpu are underutilized at 300% - 400%.
The query plan:
```
db=> explain analyze select id from person order by id limit 100;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=3821.93..3822.18 rows=100 width=8) (actual time=14.919..14.945 rows=100 loops=1)
-> Sort (cost=3821.93..4071.93 rows=100000 width=8) (actual time=14.917..14.934 rows=100 loops=1)
Sort Key: remote_scan.id
Sort Method: top-N heapsort Memory: 32kB
-> Custom Scan (Citus Adaptive) (cost=0.00..0.00 rows=100000 width=8) (actual time=14.393..14.604 rows=1600 loops=1)
Task Count: 16
Tuple data received from nodes: 13 kB
Tasks Shown: One of 16
-> Task
Tuple data received from node: 800 bytes
Node: host=mirror-citus-worker-0.mirror-citus-worker-hl.citus.svc.cluster.local port=5432 dbname=mirror_node
-> Limit (cost=0.29..2.10 rows=100 width=8) (actual time=0.188..0.216 rows=100 loops=1)
-> Index Only Scan using person_id_idx_115679 on person_115679 person (cost=0.29..565.64 rows=31237 width=8) (actual time=0.187..0.204 rows=100 loops=1)
Heap Fetches: 0
Planning Time: 0.175 ms
Execution Time: 0.272 ms
Planning Time: 0.433 ms
Execution Time: 14.994 ms
(18 rows)
```
We suspect what's overloading the coordinator is the top-N heapsort to find the 100 rows from the 1600 rows gathered from the shards. If so, the straightforward solution without considering constraints is use a better algorithm to pick the first 100 from the 16 already sorted array. And a further optimization can be, pick the first 100 from the `N` sorted array on a worker node and thus each worker node only sends 100 rows to the coordinator for the final sort and limit.
Contributor guide
Assessment
This issue has not been assessed yet.