apache / apache/gluten

[VL] Gluten reads Iceberg columns by name instead of id, causing incorrect data when column is re-added

Open
#9,528 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

When an Iceberg table column is dropped and then re-added with the same name, Iceberg assigns a new unique column ID to the re-added column. Iceberg relies on column IDs (not names) for mapping data files to schema definitions. As a result, historical data files written before the column was re-added do not contain data for the new column ID, and Iceberg correctly interprets it as null.

However, Gluten currently resolves columns by name, ignoring the column ID. This causes it to incorrectly read data from old columns with the same name but different IDs, leading to incorrect or misleading query results.

### Gluten version

main

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating Gluten's Velox path for resolving Iceberg columns, then reproduce a drop-and-re-add case to compare column IDs with the returned data; done means the re-added column reads null from historical files rather than data from the old column.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.