citusdata / citusdata/citus

Multi-shard range search query performance issue

Open
#6,746 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

We observe multi-shard range search query performance issue in our dev citus cluster.

The cluster has one coorindator and 3 workers, all configured with 8 CPUs and 30G memory. To demonstrate the performance issue, we created a simple table as follows (note the table is set to have 16 shards)

```sql
create table person (id bigint not null, name text not null);
select create_distributed_table('person', 'id');
create index on person (id);
```

To test the performance, we create a REST server with a simple endpoint to get the first 100 persons from the database, essentially every request runs the query `select * from person order by id limit 100`.

The number of requests handled per second measured is about 2000/s. Metrics show the coordinator is kept busy with 800% cpu usage and the worker nodes cpu are underutilized at 300% - 400%.

The query plan:

```
db=> explain analyze select id from person order by id limit 100;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=3821.93..3822.18 rows=100 width=8) (actual time=14.919..14.945 rows=100 loops=1)
-> Sort (cost=3821.93..4071.93 rows=100000 width=8) (actual time=14.917..14.934 rows=100 loops=1)
Sort Key: remote_scan.id
Sort Method: top-N heapsort Memory: 32kB
-> Custom Scan (Citus Adaptive) (cost=0.00..0.00 rows=100000 width=8) (actual time=14.393..14.604 rows=1600 loops=1)
Task Count: 16
Tuple data received from nodes: 13 kB
Tasks Shown: One of 16
-> Task
Tuple data received from node: 800 bytes
Node: host=mirror-citus-worker-0.mirror-citus-worker-hl.citus.svc.cluster.local port=5432 dbname=mirror_node
-> Limit (cost=0.29..2.10 rows=100 width=8) (actual time=0.188..0.216 rows=100 loops=1)
-> Index Only Scan using person_id_idx_115679 on person_115679 person (cost=0.29..565.64 rows=31237 width=8) (actual time=0.187..0.204 rows=100 loops=1)
Heap Fetches: 0
Planning Time: 0.175 ms
Execution Time: 0.272 ms
Planning Time: 0.433 ms
Execution Time: 14.994 ms
(18 rows)
```

We suspect what's overloading the coordinator is the top-N heapsort to find the 100 rows from the 1600 rows gathered from the shards. If so, the straightforward solution without considering constraints is use a better algorithm to pick the first 100 from the 16 already sorted array. And a further optimization can be, pick the first 100 from the `N` sorted array on a worker node and thus each worker node only sends 100 rows to the coordinator for the final sort and limit.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.