cockroachdb / cockroachdb/cockroach
sql: hoisting of many uncorrelated subqueries can use large amount of memory
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
In 23.2 we enabled inlining of uncorrelated equality subqueries by default. Usually this results in a more optimized plan, due to additional optimization opportunities on the inlined subqueries. If many subqueries are inlined, however, we can end up with a very deep plan using many crossjoiners, which can use a very large amount of memory.
Here's an example query with 23 subqueries. With `optimizer_hoist_uncorrelated_equality_subqueries = on` the query uses over 30 MiB of memory. With `optimizer_hoist_uncorrelated_equality_subqueries = off` the query uses < 1 MiB of memory. Both executions take less than 10ms, so there's not a huge difference in execution time.
```sql
SET CLUSTER SETTING sql.stats.automatic_collection.enabled = off;
CREATE TABLE t (
a STRING NOT NULL,
b STRING NOT NULL,
c STRING NOT NULL,
PRIMARY KEY (a, b)
);
INSERT INTO t
SELECT format('%2s', i), format('%2s', i), repeat('x', 10000)
FROM generate_series(0, 99) AS s(i);
ANALYZE t;
EXPLAIN ANALYZE
SELECT *
FROM t
WHERE
(a = ' 0' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 0'))
OR (a = ' 1' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 1'))
OR (a = ' 2' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 2'))
OR (a = ' 3' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 3'))
OR (a = ' 4' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 4'))
OR (a = ' 5' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 5'))
OR (a = ' 6' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 6'))
OR (a = ' 7' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 7'))
OR (a = ' 8' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 8'))
OR (a = ' 9' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 9'))
OR (a = '10' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '10'))
OR (a = '11' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '11'))
OR (a = '12' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '12'))
OR (a = '13' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '13'))
OR (a = '14' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '14'))
OR (a = '15' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '15'))
OR (a = '16' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '16'))
OR (a = '17' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '17'))
OR (a = '18' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '18'))
OR (a = '19' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '19'))
OR (a = '20' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '20'))
OR (a = '21' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '21'))
OR (a = '22' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '22'));
SET optimizer_hoist_uncorrelated_equality_subqueries = off;
EXPLAIN ANALYZE
SELECT *
FROM t
WHERE
(a = ' 0' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 0'))
OR (a = ' 1' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 1'))
OR (a = ' 2' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 2'))
OR (a = ' 3' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 3'))
OR (a = ' 4' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 4'))
OR (a = ' 5' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 5'))
OR (a = ' 6' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 6'))
OR (a = ' 7' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 7'))
OR (a = ' 8' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 8'))
OR (a = ' 9' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 9'))
OR (a = '10' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '10'))
OR (a = '11' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '11'))
OR (a = '12' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '12'))
OR (a = '13' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '13'))
OR (a = '14' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '14'))
OR (a = '15' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '15'))
OR (a = '16' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '16'))
OR (a = '17' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '17'))
OR (a = '18' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '18'))
OR (a = '19' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '19'))
OR (a = '20' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '20'))
OR (a = '21' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '21'))
OR (a = '22' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '22'));
```
Jira issue: CRDB-47616
Contributor guide
Assessment
This issue has not been assessed yet.