[BUG] max(BIGINT) Returns Wrong Value with High Cardinality Input and Velox Engine
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
When running a GROUP BY query with max(ts) on a BIGINT column, Gluten 1.3 with the Velox backend returns an incorrect result for some inputs.
example:
```
select
uid,
max(msg_ts) as last_msg_ts
from
test_parquet_max
where uid = '710726744'
group by uid
```
excepted result:
uid: 710726744 last_msg_ts: 1749185469637
Actual result
uid: 710726744 last_msg_ts: 140485873811016
veloxplan:
```
I20250730 10:10:50.677613 28032 VeloxRuntime.cc:145] ############### Velox plan for task [Stage: 0 TID: 5] ###############
-- Project[3][expressions: (n3_2:INTEGER, hash_with_seed(42,"n1_5")), (n3_3:BIGINT, "n1_5"), (n3_4:BIGINT, "n2_1")] -> n3_2:INTEGER, n3_3:BIGINT, n3_4:BIGINT
-- Aggregation[2][PARTIAL [n1_5] n2_1 := max_partial("n1_4")] -> n1_5:BIGINT, n2_1:BIGINT
-- Project[1][expressions: (n1_4:BIGINT, "n0_0"), (n1_5:BIGINT, "n0_1")] -> n1_4:BIGINT, n1_5:BIGINT
-- TableScan[0][table: hive_table, remaining filter: (and(isnotnull("uid"),equalto("uid",710726744)))] -> n0_0:BIGINT, n0_1:BIGINT, n0_2:VARCHAR, n0_3:VARCHAR
I20250730 10:10:54.165143 28032 VeloxRuntime.cc:145] ############### Velox plan for task [Stage: 2 TID: 20] ###############
-- Aggregation[1][SINGLE [n0_0] n1_1 := max_merge_extract("n0_1")] -> n0_0:BIGINT, n1_1:BIGINT
-- ValueStream[0][] -> n0_0:BIGINT, n0_1:BIGINT
```
### Gluten version
Gluten-1.3
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start by reproducing the GROUP BY query with the Velox backend and inspect the Velox plan and logs emitted by VeloxRuntime.cc. Compare the partial max and max_merge_extract stages against the expected BIGINT result. Done means the query returns the correct maximum value for the reported high-cardinality input.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100