apache / apache/datafusion-comet

Incorrect AVG result from empty native partial buffers

Open
#5,418 0 comments 0 reactions 0 assignees View on GitHub
area:aggregation bug priority:critical
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

An empty native scalar AVG partial currently exports `(sum = NULL, count = 0)`. Spark initializes AVG buffers to `(sum = 0, count = 0)` and adds partial sums without coalescing nulls. In the supported native-partial / Spark-final path for non-decimal AVG, an empty partial can therefore turn a nonempty query result into `NULL`.

The same scalar-state distinction matters for decimal AVG: a legitimate empty buffer must be distinguishable from an overflow buffer. A fix must keep the final result of an empty AVG as `NULL` and preserve decimal overflow behavior.

### Steps to reproduce

With Comet installed and enabled in a fresh local Spark session (for example, `local[4]`), create four separate Parquet scan partitions and filter out all rows in three of them:

```python
import tempfile

for key, value in {
"spark.sql.adaptive.enabled": "false",
"spark.sql.files.maxPartitionBytes": "1048576",
"spark.sql.parquet.filterPushdown": "false",
"spark.comet.scan.enabled": "false",
"spark.comet.convert.parquet.enabled": "true",
"spark.comet.shuffle.enabled": "false",
"spark.comet.testing.aggregate.finalMode.enabled": "false",
}.items():
spark.conf.set(key, value)

with tempfile.TemporaryDirectory() as tmp:
path = f"{tmp}/data"
(spark.range(8, numPartitions=4)
.selectExpr("id", "cast(id + 1 as int) quantity")
.write.parquet(path))
spark.read.parquet(path).createOrReplaceTempView("avg_partial_probe")
spark.sql("SELECT avg(quantity) FROM avg_partial_probe WHERE id = 1").show()
```

The final-mode testing switch isolates the native-partial / Spark-final boundary. The query has a native partial aggregate and a Spark final aggregate, but returns `NULL` instead of `2.0`.

### Expected behavior

The result is `2.0`, matching Spark. If the filter removes every row, the final result remains `NULL`.

### Additional context

Reproduced against Apache Comet `main` at `2699f59b71788e17a2714910e166a3f83deed937` using Spark 4.0.4. This is a partial-buffer representation problem, not a request to enable mixed execution for decimal AVG.

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the provided Spark configuration and four-partition Parquet query, then trace the native-partial / Spark-final AVG boundary. Done means a nonempty filtered result returns 2.0, an all-empty result remains NULL, and decimal AVG overflow behavior is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.