apache / apache/gluten

[VL](Gluten 1.5.0, Iceberg) Query produces different result compared with Spark

Open
#11,319 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

On Gluten 1.5.0 Velox Iceberg:

select
count(*) as failures,
count(*) != 0 as should_warn,
count(*) != 0 as should_error
from (
with validation_errors as (
select
ingestion_id,
count(*) as cnt
from (
select *
from schema.table.`branch_0.1.12-pd`
where ingestion_ts > date_sub(current_timestamp(), 3)
) dbt_subquery
group by ingestion_id
having count(*) > 1
or ingestion_id is null
)
select * from validation_errors
)

outputs:

failures | should_warn | should_error
-------------------------------------
8950 | true | true

where on vanilla spark 3.5.6:

failures | should_warn | should_error
-------------------------------------
0 | false | false

Info on columns:

ingestion_id: binary
ingestion_ts: timestamp_ntz

purpose of query is a DQ check to see if there are any duplicate ingestion IDs.

### Gluten version

1.5.0
### Spark version

3.5.6
### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

No source files, tests, logs, or system details are provided. Start by reproducing the supplied SQL against Gluten 1.5.0 with Velox and Iceberg, then compare it with Spark 3.5.6 using the stated binary ingestion_id and timestamp_ntz columns; done means identifying why the duplicate count differs or documenting the missing reproduction details.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.