[VL](Gluten 1.5.0, Iceberg) Query produces different result compared with Spark
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
On Gluten 1.5.0 Velox Iceberg:
select
count(*) as failures,
count(*) != 0 as should_warn,
count(*) != 0 as should_error
from (
with validation_errors as (
select
ingestion_id,
count(*) as cnt
from (
select *
from schema.table.`branch_0.1.12-pd`
where ingestion_ts > date_sub(current_timestamp(), 3)
) dbt_subquery
group by ingestion_id
having count(*) > 1
or ingestion_id is null
)
select * from validation_errors
)
outputs:
failures | should_warn | should_error
-------------------------------------
8950 | true | true
where on vanilla spark 3.5.6:
failures | should_warn | should_error
-------------------------------------
0 | false | false
Info on columns:
ingestion_id: binary
ingestion_ts: timestamp_ntz
purpose of query is a DQ check to see if there are any duplicate ingestion IDs.
### Gluten version
1.5.0
### Spark version
3.5.6
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
No source files, tests, logs, or system details are provided. Start by reproducing the supplied SQL against Gluten 1.5.0 with Velox and Iceberg, then compare it with Spark 3.5.6 using the stated binary ingestion_id and timestamp_ntz columns; done means identifying why the duplicate count differs or documenting the missing reproduction details.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100