apache / apache/doris

[Bug] MultiDistinct can OOM on group-by with high-NDV or unknown-statistics distinct arguments

Open
#66,044 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

master / 4.0

### What's Wrong?

The `MultiDistinct` strategy keeps a per-group hash set of every distinct value on a single node. For a group-by query whose distinct argument is near-unique (or whose statistics are unknown and could be near-unique), one node can end up holding a whole group's value set and OOM, e.g.:

```sql
select count(distinct payment_id), ... from t group by merchant_id
```

The no-group-by branch of `DistinctAggStrategySelector.shouldUseMultiDistinct` already guards against this by checking distinct-argument NDV and routing unknown-statistics cases to the CTE split strategy. The group-by branch does not: it neither checks distinct-argument NDV, nor treats unknown distinct-argument statistics as risky, and on unknown group-by statistics it unconditionally chooses `MultiDistinct`.

### What You Expected?

For a group-by query whose distinct argument is near-unique relative to input rows, or whose distinct argument (or a slot it reads) has unknown statistics, the planner should route to the CTE split strategy (which redistributes on the distinct key) to avoid the single-node OOM — mirroring the no-group-by branch.

### How to Reproduce?

Run a group-by aggregation with one or more `count(distinct ...)` over a near-unique column (or a column wrapped in `if(cond, col, null)` where `col` is unanalyzed), and observe that `MultiDistinct` is chosen and can OOM under a low-cardinality group-by key.

### Anything Else?

Fix (group-by branch of `shouldUseMultiDistinct`), matching the no-group-by branch:
1. `hasHighNdvDistinctArgument`: force CTE split when any distinct argument's estimated NDV >= `row * MID_CARDINALITY_THRESHOLD`.
2. `hasUnknownNdvDistinctArgument`: force CTE split when a distinct argument's estimate is unknown, or any input slot it reads has absent/unknown statistics (the per-slot scan is required because `ExpressionEstimation.visitIf` fabricates a non-unknown NDV for `if(cond, col, null)` even when `col` is unanalyzed).
3. When group-by statistics are unknown (reached only when every distinct argument is confirmed low-NDV), prefer CTE split for a single group-by key and keep `MultiDistinct` for >= 2 keys.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start at DistinctAggStrategySelector.shouldUseMultiDistinct and compare the group-by branch with the no-group-by branch. Trace how distinct-argument and group-by statistics are estimated, then verify that high or unknown distinct NDV selects CTE split, while unknown group-by statistics select CTE split for one key and preserve MultiDistinct for two or more keys.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
57/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.