apache / apache/auron

Support multiple count distinct

Open
#1,524 2 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 14h
Merged PRs (30d)
22

Description

**Is your feature request related to a problem? Please describe.**

```
25/10/26 18:24:46 WARN AuronConverters: Falling back exec: HashAggregateExec: assertion failed: partial AggregateExec is not native
25/10/26 18:24:57 WARN AuronConverters: Falling back exec: HashAggregateExec: assertion failed
25/10/26 18:24:59 WARN AuronConverters: Falling back exec: HashAggregateExec: assertion failed: partial AggregateExec is not native
```

```sql
SELECT
region,
year,
COUNT(DISTINCT product) AS distinct_product_count,
COUNT(DISTINCT region) AS distinct_region_count
FROM (
SELECT
CASE WHEN id % 2 = 0 THEN 'Region1' ELSE 'Region2' END AS region,
CASE WHEN id % 3 = 0 THEN 'ProductA' ELSE 'ProductB' END AS product,
2023 AS year,
id * 100 AS sales_amount
FROM RANGE(10)
) AS sales
GROUP BY region, year;
```

Because Spark rule RewriteDistinctAggregates will rewrite multiple COUNT DISTINCT and add gid filter, Auron convertMoreAggregateExpr restricts the filter to be empty.

https://github.com/apache/auron/blob/3eb0e7353d0e280ddc7ae3dab77b8206c31dfb76/spark-extension-shims-spark/src/main/scala/org/apache/spark/sql/auron/ShimsImpl.scala#L558-L559

https://github.com/apache/spark/blob/65755bde3cb8b09bfaa8776fe6b4a182fd74ee0b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala#L249-L251

---
[SPARK-34882][SQL] Replace if with filter clause in RewriteDistinctAggregates
https://issues.apache.org/jira/browse/SPARK-34882
Fix Version/s: 3.2.0
**Describe the solution you'd like**

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.