apache / apache/gluten

[VL] GlutenDataFrameAggregateSuite NaN and -0.0 in grouping expressions

Open
#11,234 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

```

Results do not match for query:
Timezone: sun.util.calendar.ZoneInfo[id="America/Los_Angeles",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=America/Los_Angeles,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]]
Timezone Env:

== Parsed Logical Plan ==
'Aggregate [f#12325], [f#12325, 'count(1) AS count#12326]
+- Project [value#12321 AS f#12325]
+- LocalRelation [value#12321]

== Analyzed Logical Plan ==
f: float, count: bigint
Aggregate [f#12325], [f#12325, count(1) AS count#12326L]
+- Project [value#12321 AS f#12325]
+- LocalRelation [value#12321]

== Optimized Logical Plan ==
Aggregate [f#12325], [f#12325, count(1) AS count#12326L]
+- Project [value#12321 AS f#12325]
+- LocalRelation [value#12321]

== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=true
+- == Final Plan ==
ResultQueryStage 1
+- VeloxColumnarToRow
+- ^(494) HashAggregateTransformer(keys=[f#12325], functions=[count(1)], isStreamingAgg=false, output=[f#12325, count#12326L])
+- ^(494) InputIteratorTransformer[f#12325, count#12330L]
+- ShuffleQueryStage 0
+- ColumnarExchange hashpartitioning(f#12325, 1), ENSURE_REQUIREMENTS, [f#12325, count#12330L], [plan_id=59771], [shuffle_writer_type=hash], [output=[f#12325: float, count#12330: bigint]]
+- VeloxResizeBatches 1024, 2147483647
+- ^(493) ProjectExecTransformer [hash(f#12325, 42) AS hash_partition_key#12344, f#12325, count#12330L]
+- ^(493) FlushableHashAggregateTransformer(keys=[f#12325], functions=[partial_count(1)], isStreamingAgg=false, output=[f#12325, count#12330L])
+- ^(493) ProjectExecTransformer [knownfloatingpointnormalized(normalizenanandzero(value#12321)) AS f#12325]
+- ^(493) InputIteratorTransformer[value#12321]
+- RowToVeloxColumnar
+- LocalTableScan [value#12321]
+- == Initial Plan ==
HashAggregate(keys=[f#12325], functions=[count(1)], output=[f#12325, count#12326L])
+- Exchange hashpartitioning(f#12325, 1), ENSURE_REQUIREMENTS, [plan_id=59358]
+- HashAggregate(keys=[knownfloatingpointnormalized(normalizenanandzero(f#12325)) AS f#12325], functions=[partial_count(1)], output=[f#12325, count#12330L])
+- Project [value#12321 AS f#12325]
+- LocalTableScan [value#12321]

== Results ==

== Results ==
!== Correct Answer - 2 == == Gluten Answer - 2 ==
!struct<> struct
[0.0,2] [0.0,2]
[NaN,2] [NaN,2]
```

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start with GlutenDataFrameAggregateSuite and the grouping-expression case for NaN and -0.0. Run the relevant Spark/Velox aggregation test and compare its output with the expected result shown in the issue; done means the grouping results match without regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.