cockroachdb / cockroachdb/cockroach

opt: do not create column statistics for unfiltered columns

Open
#121,311 0 comments 0 reactions 0 assignees View on GitHub
C-enhancement O-support P-3 T-sql-queries
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When estimating row counts, the statistics builder builds column statistics for all columns in base relations that have collected stats, even if they are never used to estimate row counts in any parent expressions. This can cause excessive and unnecessary heap allocations, especially when base relations have many columns. Large column IDs cause allocations when creating the `opt.ColSet` here:

https://github.com/cockroachdb/cockroach/blob/579d42e9aca2e51548d270f7e5f4557b31ff502e/pkg/sql/opt/memo/statistics_builder.go#L675-L679

And when adding to the `props.ColStatsMap` here:

https://github.com/cockroachdb/cockroach/blob/579d42e9aca2e51548d270f7e5f4557b31ff502e/pkg/sql/opt/memo/statistics_builder.go#L716

I've seen these two account for 10% of allocated objects and 8.5% of allocated space in some workloads.

I believe we can eliminate most of these allocations by only building column statistics for columns that are filtered in parent expressions. This will be a bit tricky to do while also caching table statistics in base relations, as is currently done:

https://github.com/cockroachdb/cockroach/blob/579d42e9aca2e51548d270f7e5f4557b31ff502e/pkg/sql/opt/memo/statistics_builder.go#L610-L615

More discussion can be found in https://cockroachlabs.slack.com/archives/G01R3EYPQKA/p1711577752592149

Jira issue: CRDB-37179

Epic CRDB-41124

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.