apache / apache/datafusion-comet

[Enhancement] Enable mixed partial/final execution for approx_count_distinct (HyperLogLogPlusPlus)

Open
#4,820 1 comment 0 reactions 1 assignee Claimed by @michaelmitchell-bit View on GitHub
area:aggregation enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Background

The initial `approx_count_distinct` support (PR #4819) stores its HyperLogLog++ registers in Spark's exact packed-`Long` buffer layout (`numWords` `Long` columns, 10 six-bit registers per word), so Comet's partial-aggregation state is byte-identical to Spark's `HyperLogLogPlusPlus.aggBufferSchema`.

`CometApproxCountDistinct` currently leaves `supportsMixedPartialFinal` at the default `false`, which means that whenever a plan has an `approx_count_distinct` at a Comet/Spark boundary, `allAggsSupportMixedExecution` forces **both** the partial and final aggregate onto the same engine.

### Proposal

Because the intermediate buffer format now matches Spark exactly, `supportsMixedPartialFinal` should be safe to set to `true` (as `CometMin`, `CometMax`, and the bitwise aggregates already do). This would let Comet accelerate the partial aggregate even when the final falls back to Spark (and vice versa), broadening native coverage.

### Work required

- Set `supportsMixedPartialFinal = true` in `CometApproxCountDistinct`.
- Add a partial-merge test (see `spark/src/test/resources/sql-tests/expressions/aggregate/partial_merge.sql`) that exercises Comet-partial + Spark-final and Spark-partial + Comet-final, confirming the result stays bit-identical to Spark.

This was deferred from PR #4819 because it needs interop verification rather than being a trivial one-line change.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.