apache / apache/datafusion-comet
Incorrect AVG result from empty native partial buffers
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
An empty native scalar AVG partial currently exports `(sum = NULL, count = 0)`. Spark initializes AVG buffers to `(sum = 0, count = 0)` and adds partial sums without coalescing nulls. In the supported native-partial / Spark-final path for non-decimal AVG, an empty partial can therefore turn a nonempty query result into `NULL`.
The same scalar-state distinction matters for decimal AVG: a legitimate empty buffer must be distinguishable from an overflow buffer. A fix must keep the final result of an empty AVG as `NULL` and preserve decimal overflow behavior.
### Steps to reproduce
With Comet installed and enabled in a fresh local Spark session (for example, `local[4]`), create four separate Parquet scan partitions and filter out all rows in three of them:
```python
import tempfile
for key, value in {
"spark.sql.adaptive.enabled": "false",
"spark.sql.files.maxPartitionBytes": "1048576",
"spark.sql.parquet.filterPushdown": "false",
"spark.comet.scan.enabled": "false",
"spark.comet.convert.parquet.enabled": "true",
"spark.comet.shuffle.enabled": "false",
"spark.comet.testing.aggregate.finalMode.enabled": "false",
}.items():
spark.conf.set(key, value)
with tempfile.TemporaryDirectory() as tmp:
path = f"{tmp}/data"
(spark.range(8, numPartitions=4)
.selectExpr("id", "cast(id + 1 as int) quantity")
.write.parquet(path))
spark.read.parquet(path).createOrReplaceTempView("avg_partial_probe")
spark.sql("SELECT avg(quantity) FROM avg_partial_probe WHERE id = 1").show()
```
The final-mode testing switch isolates the native-partial / Spark-final boundary. The query has a native partial aggregate and a Spark final aggregate, but returns `NULL` instead of `2.0`.
### Expected behavior
The result is `2.0`, matching Spark. If the filter removes every row, the final result remains `NULL`.
### Additional context
Reproduced against Apache Comet `main` at `2699f59b71788e17a2714910e166a3f83deed937` using Spark 4.0.4. This is a partial-buffer representation problem, not a request to enable mixed execution for decimal AVG.
Contributor guide
Research direction
Reproduce the issue with the provided Spark configuration and four-partition Parquet query, then trace the native-partial / Spark-final AVG boundary. Done means a nonempty filtered result returns 2.0, an all-empty result remains NULL, and decimal AVG overflow behavior is preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100