Volatile functions in subqueries on reference tables incorrectly re-evaluated
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
When you have a subquery like `(SELECT *, random() FROM reference_table)`, the PostgreSQL interpretation is that the random() is evaluated once for per row in reference table, which means that if the subquery is used in a join and a row matches multiple rows on the other side of the join, the value of random() will be the same across all joined rows.
In Citus, the value may or may not be re-evaluated after the join, because random may result in different values when querying different shards.
```sql
CREATE TABLE test (x int, y int);
CREATE TABLE ref (LIKE test);
INSERT INTO test VALUES (1,1);
INSERT INTO test VALUES (2,1);
INSERT INTO ref VALUES (1,1);
-- local table: same value in all joined rows
SELECT * FROM test JOIN (SELECT y, random() FROM ref) r USING (y);
y | x | random
---+---+-------------------
1 | 1 | 0.370642014313489
1 | 2 | 0.370642014313489
(2 rows)
SELECT create_distributed_table('test','x');
SELECT create_reference_table('ref');
-- distributed table: different value in all joined rows
SELECT * FROM test JOIN (SELECT y, random() FROM ref) r USING (y);
y | x | random
---+---+-------------------
1 | 1 | 0.527601474896073
1 | 2 | 0.262537915725261
(2 rows)
Time: 109.718 ms
```
Locally, the query does follow PostgreSQL semantics, which can lead to a strange mix of identical and differing values.
```
INSERT INTO test VALUES (2,1);
SELECT * FROM test JOIN (SELECT y, random() FROM ref) r USING (y);
y | x | random
---+---+-------------------
1 | 1 | 0.69018074683845
1 | 2 | 0.783294859807938
1 | 2 | 0.783294859807938
(3 rows)
```
We did anticipate this case for subqueries without FROM:
```sql
SELECT * FROM test JOIN (SELECT 1 AS y, random()) r USING (y);
ERROR: cannot push down this subquery
DETAIL: Subqueries without a FROM clause can only contain immutable functions
```
but not for subqueries with an immutable function RTE:
```sql
SELECT * FROM test JOIN (SELECT 1 AS y, random() FROM generate_series(1,1) s) r USING (y);
y | x | random
---+---+-------------------
1 | 1 | 0.437155081424862
1 | 2 | 0.583905343897641
1 | 2 | 0.583905343897641
(3 rows)
```
We should probably disallow this case, and then plan it recursively after #1804.
For subqueries on reference tables, we could also consider always planning them recursively (after #1804) to avoid using slightly different versions of the reference table in each join, which is probably a more severe problem than random being re-evaluated.
Contributor guide
Assessment
This issue has not been assessed yet.