Unnecessary intermediate result broadcasts
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
https://github.com/citusdata/citus/commit/d82f3e9406da2f9c872e7fa354d27033ca244e5c prunes intermediate results so they are only broadcast to the nodes that are necessary.
Take the following query, which is taken from https://github.com/citusdata/citus/blob/d82f3e9406da2f9c872e7fa354d27033ca244e5c/src/test/regress/sql/intermediate_result_pruning.sql, in which `table_1` and `table_2` are distributed tables, and `ref_table` is a reference table:
```sql
WITH some_values_1 AS
(SELECT key, random() FROM table_1 WHERE value IN ('3', '4')),
some_values_2 AS
(SELECT key, random() FROM some_values_1 JOIN table_2 USING (key) WHERE key = 1),
some_values_3 AS
(SELECT key FROM (some_values_2 JOIN table_2 USING (key)) JOIN some_values_1 USING (key))
SELECT * FROM some_values_3 JOIN ref_table ON (true);
```
the results of the `some_values_3` CTE is broadcast to all nodes that have reference table placement, but it is not necessary and we can broadcast it to only one of the nodes.
Specifically, when I add the coordinator to pg_dist_node using `SELECT master_add_node(..., groupId => 0)`, this intermediate result is also broadcast to the coordinator.
Contributor guide
Assessment
This issue has not been assessed yet.