apache / apache/gluten

Iceberg's parquets do not have field_id (schema evolution is broken)

Open
#10,394 2 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Backend

VL (Velox)

### Bug description

[Iceberg spec](https://iceberg.apache.org/spec/#parquet) requires field_ids are set:

> Column IDs are required to be stored as [field IDs](http://github.com/apache/parquet-format/blob/40699d05bd24181de6b1457babbee2c16dce3803/src/main/thrift/parquet.thrift#L459) on the parquet schema.

As I could understand the actual column ids from Iceberg schema are not passed here. So they cannot be written in Velox later.

https://github.com/apache/incubator-gluten/blob/2ec3ba751821d5e09a4da630c2b55e8a1a3ccb1b/cpp/velox/compute/VeloxRuntime.cc#L232

It looks like it would be possible to pass the ids in Velox part through IcebergColumnHandle after this [PR](https://github.com/facebookincubator/velox/pull/14272/files#diff-9113b137842f261f316379c01b28852014c7d6ee19c0d74c0f9f757f5ed34ce3R23)

Am I right that there's no info about actual Iceberg column_ids in `Java_org_apache_gluten_execution_IcebergWriteJniWrapper_init` right now?

### Gluten version

main branch

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start at cpp/velox/compute/VeloxRuntime.cc around line 232 and inspect Java_org_apache_gluten_execution_IcebergWriteJniWrapper_init to determine whether Iceberg column IDs are available there. Read the referenced IcebergColumnHandle change in the Velox pull request; the work is complete when Iceberg IDs can be carried into Velox and written as Parquet field IDs for schema evolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.