Iceberg's parquets do not have field_id (schema evolution is broken)
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Backend
VL (Velox)
### Bug description
[Iceberg spec](https://iceberg.apache.org/spec/#parquet) requires field_ids are set:
> Column IDs are required to be stored as [field IDs](http://github.com/apache/parquet-format/blob/40699d05bd24181de6b1457babbee2c16dce3803/src/main/thrift/parquet.thrift#L459) on the parquet schema.
As I could understand the actual column ids from Iceberg schema are not passed here. So they cannot be written in Velox later.
https://github.com/apache/incubator-gluten/blob/2ec3ba751821d5e09a4da630c2b55e8a1a3ccb1b/cpp/velox/compute/VeloxRuntime.cc#L232
It looks like it would be possible to pass the ids in Velox part through IcebergColumnHandle after this [PR](https://github.com/facebookincubator/velox/pull/14272/files#diff-9113b137842f261f316379c01b28852014c7d6ee19c0d74c0f9f757f5ed34ce3R23)
Am I right that there's no info about actual Iceberg column_ids in `Java_org_apache_gluten_execution_IcebergWriteJniWrapper_init` right now?
### Gluten version
main branch
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start at cpp/velox/compute/VeloxRuntime.cc around line 232 and inspect Java_org_apache_gluten_execution_IcebergWriteJniWrapper_init to determine whether Iceberg column IDs are available there. Read the referenced IcebergColumnHandle change in the Velox pull request; the work is complete when Iceberg IDs can be carried into Velox and written as Parquet field IDs for schema evolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100