[VL] GlutenDataFrameAggregateSuite NaN and -0.0 in grouping expressions
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
```
Results do not match for query:
Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
Timezone Env:
== Parsed Logical Plan ==
'Aggregate [f#12325], [f#12325, 'count(1) AS count#12326]
+- Project [value#12321 AS f#12325]
+- LocalRelation [value#12321]
== Analyzed Logical Plan ==
f: float, count: bigint
Aggregate [f#12325], [f#12325, count(1) AS count#12326L]
+- Project [value#12321 AS f#12325]
+- LocalRelation [value#12321]
== Optimized Logical Plan ==
Aggregate [f#12325], [f#12325, count(1) AS count#12326L]
+- Project [value#12321 AS f#12325]
+- LocalRelation [value#12321]
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
ResultQueryStage 1
+- VeloxColumnarToRow
+- ^(494) HashAggregateTransformer(keys=[f#12325], functions=[count(1)], isStreamingAgg=false, output=[f#12325, count#12326L])
+- ^(494) InputIteratorTransformer[f#12325, count#12330L]
+- ShuffleQueryStage 0
+- ColumnarExchange hashpartitioning(f#12325, 1), ENSURE_REQUIREMENTS, [f#12325, count#12330L], [plan_id=59771], [shuffle_writer_type=hash], [output=[f#12325: float, count#12330: bigint]]
+- VeloxResizeBatches 1024, 2147483647
+- ^(493) ProjectExecTransformer [hash(f#12325, 42) AS hash_partition_key#12344, f#12325, count#12330L]
+- ^(493) FlushableHashAggregateTransformer(keys=[f#12325], functions=[partial_count(1)], isStreamingAgg=false, output=[f#12325, count#12330L])
+- ^(493) ProjectExecTransformer [knownfloatingpointnormalized(normalizenanandzero(value#12321)) AS f#12325]
+- ^(493) InputIteratorTransformer[value#12321]
+- RowToVeloxColumnar
+- LocalTableScan [value#12321]
+- == Initial Plan ==
HashAggregate(keys=[f#12325], functions=[count(1)], output=[f#12325, count#12326L])
+- Exchange hashpartitioning(f#12325, 1), ENSURE_REQUIREMENTS, [plan_id=59358]
+- HashAggregate(keys=[knownfloatingpointnormalized(normalizenanandzero(f#12325)) AS f#12325], functions=[partial_count(1)], output=[f#12325, count#12330L])
+- Project [value#12321 AS f#12325]
+- LocalTableScan [value#12321]
== Results ==
== Results ==
!== Correct Answer - 2 == == Gluten Answer - 2 ==
!struct<> struct
[0.0,2] [0.0,2]
[NaN,2] [NaN,2]
```
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start with GlutenDataFrameAggregateSuite and the grouping-expression case for NaN and -0.0. Run the relevant Spark/Velox aggregation test and compare its output with the expected result shown in the issue; done means the grouping results match without regressions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100