citusdata / citusdata/citus

Unnecessary intermediate result broadcasts

Open
#3,217 4 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

https://github.com/citusdata/citus/commit/d82f3e9406da2f9c872e7fa354d27033ca244e5c prunes intermediate results so they are only broadcast to the nodes that are necessary.

Take the following query, which is taken from https://github.com/citusdata/citus/blob/d82f3e9406da2f9c872e7fa354d27033ca244e5c/src/test/regress/sql/intermediate_result_pruning.sql, in which `table_1` and `table_2` are distributed tables, and `ref_table` is a reference table:

```sql
WITH some_values_1 AS
(SELECT key, random() FROM table_1 WHERE value IN ('3', '4')),
some_values_2 AS
(SELECT key, random() FROM some_values_1 JOIN table_2 USING (key) WHERE key = 1),
some_values_3 AS
(SELECT key FROM (some_values_2 JOIN table_2 USING (key)) JOIN some_values_1 USING (key))
SELECT * FROM some_values_3 JOIN ref_table ON (true);
```

the results of the `some_values_3` CTE is broadcast to all nodes that have reference table placement, but it is not necessary and we can broadcast it to only one of the nodes.

Specifically, when I add the coordinator to pg_dist_node using `SELECT master_add_node(..., groupId => 0)`, this intermediate result is also broadcast to the coordinator.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.