[Bug] MultiDistinct can OOM on group-by with high-NDV or unknown-statistics distinct arguments
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
master / 4.0
### What's Wrong?
The `MultiDistinct` strategy keeps a per-group hash set of every distinct value on a single node. For a group-by query whose distinct argument is near-unique (or whose statistics are unknown and could be near-unique), one node can end up holding a whole group's value set and OOM, e.g.:
```sql
select count(distinct payment_id), ... from t group by merchant_id
```
The no-group-by branch of `DistinctAggStrategySelector.shouldUseMultiDistinct` already guards against this by checking distinct-argument NDV and routing unknown-statistics cases to the CTE split strategy. The group-by branch does not: it neither checks distinct-argument NDV, nor treats unknown distinct-argument statistics as risky, and on unknown group-by statistics it unconditionally chooses `MultiDistinct`.
### What You Expected?
For a group-by query whose distinct argument is near-unique relative to input rows, or whose distinct argument (or a slot it reads) has unknown statistics, the planner should route to the CTE split strategy (which redistributes on the distinct key) to avoid the single-node OOM — mirroring the no-group-by branch.
### How to Reproduce?
Run a group-by aggregation with one or more `count(distinct ...)` over a near-unique column (or a column wrapped in `if(cond, col, null)` where `col` is unanalyzed), and observe that `MultiDistinct` is chosen and can OOM under a low-cardinality group-by key.
### Anything Else?
Fix (group-by branch of `shouldUseMultiDistinct`), matching the no-group-by branch:
1. `hasHighNdvDistinctArgument`: force CTE split when any distinct argument's estimated NDV >= `row * MID_CARDINALITY_THRESHOLD`.
2. `hasUnknownNdvDistinctArgument`: force CTE split when a distinct argument's estimate is unknown, or any input slot it reads has absent/unknown statistics (the per-slot scan is required because `ExpressionEstimation.visitIf` fabricates a non-unknown NDV for `if(cond, col, null)` even when `col` is unanalyzed).
3. When group-by statistics are unknown (reached only when every distinct argument is confirmed low-NDV), prefer CTE split for a single group-by key and keep `MultiDistinct` for >= 2 keys.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start at DistinctAggStrategySelector.shouldUseMultiDistinct and compare the group-by branch with the no-group-by branch. Trace how distinct-argument and group-by statistics are estimated, then verify that high or unknown distinct NDV selects CTE split, while unknown group-by statistics select CTE split for one key and preserve MultiDistinct for two or more keys.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 57/100