cockroachdb / cockroachdb/cockroach
opt: do not create column statistics for unfiltered columns
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
When estimating row counts, the statistics builder builds column statistics for all columns in base relations that have collected stats, even if they are never used to estimate row counts in any parent expressions. This can cause excessive and unnecessary heap allocations, especially when base relations have many columns. Large column IDs cause allocations when creating the `opt.ColSet` here:
https://github.com/cockroachdb/cockroach/blob/579d42e9aca2e51548d270f7e5f4557b31ff502e/pkg/sql/opt/memo/statistics_builder.go#L675-L679
And when adding to the `props.ColStatsMap` here:
https://github.com/cockroachdb/cockroach/blob/579d42e9aca2e51548d270f7e5f4557b31ff502e/pkg/sql/opt/memo/statistics_builder.go#L716
I've seen these two account for 10% of allocated objects and 8.5% of allocated space in some workloads.
I believe we can eliminate most of these allocations by only building column statistics for columns that are filtered in parent expressions. This will be a bit tricky to do while also caching table statistics in base relations, as is currently done:
https://github.com/cockroachdb/cockroach/blob/579d42e9aca2e51548d270f7e5f4557b31ff502e/pkg/sql/opt/memo/statistics_builder.go#L610-L615
More discussion can be found in https://cockroachlabs.slack.com/archives/G01R3EYPQKA/p1711577752592149
Jira issue: CRDB-37179
Epic CRDB-41124
Contributor guide
Assessment
This issue has not been assessed yet.