cockroachdb / cockroachdb/cockroach
sql: OOM risk of EXPLAIN ANALYZE (DEBUG) of statement with large lookup or index join
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
At the end of statement diagnostics bundle collection of a query with a large lookup or index join, there can be a large spike in memory usage. Sometimes this is enough to OOM a node. Here's a demonstration using a real cluster:
```sql
-- create a table with a secondary index and at least one leaseholder range per node
CREATE TABLE abc (a STRING, b STRING, c STRING, INDEX (a, b));
ALTER TABLE abc CONFIGURE ZONE USING range_max_bytes = 33554432, range_min_bytes = 1048576;
-- populate
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
INSERT INTO abc SELECT i::string, i::string, i::string FROM generate_series(0, 99999) s(i);
ANALYZE abc;
SHOW RANGES FROM TABLE abc;
-- use a vectorized, distributed query with 2.6m rows passing through an index join
EXPLAIN SELECT COUNT(DISTINCT c) FROM abc@abc_a_b_idx WHERE a > '2' AND b > '2' AND c > '2';
-- normal execution takes a few seconds and only has a small impact on total node memory usage
SELECT now(); SELECT COUNT(DISTINCT c) FROM abc@abc_a_b_idx WHERE a > '2' AND b > '2' AND c > '2';
SELECT now();
-- execution with analyze (statistics collection) takes 10 second and again has a small impact on total node memory usage
SELECT now(); EXPLAIN ANALYZE SELECT COUNT(DISTINCT c) FROM abc@abc_a_b_idx WHERE a > '2' AND b > '2' AND c > '2';
SELECT now();
-- execution with verbose tracing takes over 3 minutes and causes a large spike in CPU and memory usage
SELECT now(); EXPLAIN ANALYZE (DEBUG) SELECT COUNT(DISTINCT c) FROM abc@abc_a_b_idx WHERE a > '2' AND b > '2' AND c > '2';
SELECT now();
```
Here's how that looked in the metrics:
Even though the bundle said max memory usage was 67 MiB, we actually saw a spike > 1 GiB.
This is very similar to https://github.com/cockroachdb/cockroach/issues/90739 and may even be the same root cause (verbose tracing) but I wanted to document the spike in memory usage. We should try to fix this spike even if statement bundle collection takes longer than normal execution.
I believe the spike in memory on the gateway node is due to unmarshaling of this log message in traces: https://github.com/cockroachdb/cockroach/blob/190aa543fae4b1c9cea34168575e54104deb62aa/pkg/kv/kvserver/replica_evaluate.go#L550-L551
I have reproduced this on both v23.1.0 and v21.2.17.
Jira issue: CRDB-27960
Contributor guide
Assessment
This issue has not been assessed yet.