[VL] Gluten reads Iceberg columns by name instead of id, causing incorrect data when column is re-added
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
When an Iceberg table column is dropped and then re-added with the same name, Iceberg assigns a new unique column ID to the re-added column. Iceberg relies on column IDs (not names) for mapping data files to schema definitions. As a result, historical data files written before the column was re-added do not contain data for the new column ID, and Iceberg correctly interprets it as null.
However, Gluten currently resolves columns by name, ignoring the column ID. This causes it to incorrectly read data from old columns with the same name but different IDs, leading to incorrect or misleading query results.
### Gluten version
main
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by locating Gluten's Velox path for resolving Iceberg columns, then reproduce a drop-and-re-add case to compare column IDs with the returned data; done means the re-added column reads null from historical files rather than data from the old column.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100