cockroachdb / cockroachdb/cockroach

sql: hoisting of many uncorrelated subqueries can use large amount of memory

Open
#141,110 1 comment 0 reactions 0 assignees View on GitHub
A-sql-execution A-sql-optimizer branch-release-23.2 branch-release-24.1 branch-release-24.2 branch-release-24.3 branch-release-25.1 C-performance O-support P-3 T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

In 23.2 we enabled inlining of uncorrelated equality subqueries by default. Usually this results in a more optimized plan, due to additional optimization opportunities on the inlined subqueries. If many subqueries are inlined, however, we can end up with a very deep plan using many crossjoiners, which can use a very large amount of memory.

Here's an example query with 23 subqueries. With `optimizer_hoist_uncorrelated_equality_subqueries = on` the query uses over 30 MiB of memory. With `optimizer_hoist_uncorrelated_equality_subqueries = off` the query uses < 1 MiB of memory. Both executions take less than 10ms, so there's not a huge difference in execution time.

```sql
SET CLUSTER SETTING sql.stats.automatic_collection.enabled = off;

CREATE TABLE t (
a STRING NOT NULL,
b STRING NOT NULL,
c STRING NOT NULL,
PRIMARY KEY (a, b)
);

INSERT INTO t
SELECT format('%2s', i), format('%2s', i), repeat('x', 10000)
FROM generate_series(0, 99) AS s(i);

ANALYZE t;

EXPLAIN ANALYZE
SELECT *
FROM t
WHERE
(a = ' 0' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 0'))
OR (a = ' 1' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 1'))
OR (a = ' 2' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 2'))
OR (a = ' 3' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 3'))
OR (a = ' 4' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 4'))
OR (a = ' 5' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 5'))
OR (a = ' 6' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 6'))
OR (a = ' 7' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 7'))
OR (a = ' 8' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 8'))
OR (a = ' 9' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 9'))
OR (a = '10' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '10'))
OR (a = '11' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '11'))
OR (a = '12' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '12'))
OR (a = '13' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '13'))
OR (a = '14' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '14'))
OR (a = '15' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '15'))
OR (a = '16' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '16'))
OR (a = '17' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '17'))
OR (a = '18' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '18'))
OR (a = '19' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '19'))
OR (a = '20' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '20'))
OR (a = '21' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '21'))
OR (a = '22' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '22'));

SET optimizer_hoist_uncorrelated_equality_subqueries = off;

EXPLAIN ANALYZE
SELECT *
FROM t
WHERE
(a = ' 0' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 0'))
OR (a = ' 1' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 1'))
OR (a = ' 2' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 2'))
OR (a = ' 3' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 3'))
OR (a = ' 4' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 4'))
OR (a = ' 5' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 5'))
OR (a = ' 6' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 6'))
OR (a = ' 7' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 7'))
OR (a = ' 8' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 8'))
OR (a = ' 9' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < ' 9'))
OR (a = '10' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '10'))
OR (a = '11' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '11'))
OR (a = '12' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '12'))
OR (a = '13' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '13'))
OR (a = '14' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '14'))
OR (a = '15' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '15'))
OR (a = '16' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '16'))
OR (a = '17' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '17'))
OR (a = '18' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '18'))
OR (a = '19' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '19'))
OR (a = '20' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '20'))
OR (a = '21' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '21'))
OR (a = '22' AND b = (SELECT format('%2s', count(*)) FROM t WHERE a < '22'));
```

Jira issue: CRDB-47616

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.