apache / apache/incubator-xtable

About the support for Schema Evolution (column rename)

Open
#282 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

Hi there, I tested onetable tool with a created an Iceberg Table and run a column rename.

```
spark-sql> CREATE TABLE hadoop_prod.repro_rename ( id bigint, data string) using iceberg;
Response code
Time taken: 2.269 seconds
spark-sql> insert into hadoop_prod.repro_rename values (1, "abc");
Response code
Time taken: 8.093 seconds
spark-sql> select * from hadoop_prod.repro_rename;
id data
1 abc
spark-sql> alter table hadoop_prod.repro_rename rename column id to new_id ;
Time taken: 2.255 seconds
spark-sql> select * from hadoop_prod.repro_rename;
new_id data
1 abc
```
Run onetable to convert Iceberg to Hudi and Delta, and the information about the column rename doesn't seem to be captured in the converted metadata

issue 1: the schema is still using the old one

HUDI:
```
>>df = spark.read.format("hudi").options(**hudi_options).load("MY_PATH/repro_rename")
>>> df.show(truncate=False)
+---+----+
|id |data|
+---+----+
|1 |abc |
+---+----+
```
DELTA
```
spark-sql> select * FROM delta.`MY_PATH/repro_rename` ;
id data
1 abc
```

issue 2: Iceberg's column rename is built on top of field id, I don't see any Delta/Hudi equivalence are included in the converted metadata

for Delta: "delta.columnMapping.mode":"id","delta.columnMapping.maxColumnId":"4" is missing in the delta log -- see https://docs.databricks.com/en/delta/delta-column-mapping.html for the implementation of 'delta.columnMapping.mode' = 'id'

for Hudi : hudi commit log doesn’t have id, max_column_id populated (fields in https://github.com/apache/hudi/pull/4910/files )

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Spark SQL reproduction in the issue: create an Iceberg table, rename a column, convert it to Hudi and Delta, and inspect the resulting schema and metadata. Read Delta's column-mapping documentation and the referenced Hudi change; done means the converted tables preserve the renamed column and the required field-ID metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.