Merge queries on shards that are assigned to the same connection
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
When two tasks are assigned to the same connection, we execute them one by one. If the tasks belong to the same query and simply query different shards, then we could merge them together by taking the UNION ALL of the shards.
This can be achieved by encoding the list of shard IDs in the citus_extradata_container that is normally used to pass the shard ID to the deparser:
https://github.com/citusdata/citus/blob/e0cccf7f9af7ab6818bf3b0da19df7e1159d10d1/src/backend/distributed/deparser/ruleutils_12.c#L7012
At that point, instead of appending the relation name, we append a subquery that unions all the shards and let the PostgreSQL planner on the worker figure out how to plan the query efficiently.
This would also put us in a better spot to leverage postgres parallelism to query a greater number of shards in parallel without connection establishment overhead, reduce network traffic when aggregating across a non-distribution-column, and reduce repetitive work (e.g. reference table joins).
Contributor guide
Assessment
This issue has not been assessed yet.