apache / apache/incubator-xtable
About the support for Schema Evolution (column rename)
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
Hi there, I tested onetable tool with a created an Iceberg Table and run a column rename.
```
spark-sql> CREATE TABLE hadoop_prod.repro_rename ( id bigint, data string) using iceberg;
Response code
Time taken: 2.269 seconds
spark-sql> insert into hadoop_prod.repro_rename values (1, "abc");
Response code
Time taken: 8.093 seconds
spark-sql> select * from hadoop_prod.repro_rename;
id data
1 abc
spark-sql> alter table hadoop_prod.repro_rename rename column id to new_id ;
Time taken: 2.255 seconds
spark-sql> select * from hadoop_prod.repro_rename;
new_id data
1 abc
```
Run onetable to convert Iceberg to Hudi and Delta, and the information about the column rename doesn't seem to be captured in the converted metadata
issue 1: the schema is still using the old one
HUDI:
```
>>df = spark.read.format("hudi").options(**hudi_options).load("MY_PATH/repro_rename")
>>> df.show(truncate=False)
+---+----+
|id |data|
+---+----+
|1 |abc |
+---+----+
```
DELTA
```
spark-sql> select * FROM delta.`MY_PATH/repro_rename` ;
id data
1 abc
```
issue 2: Iceberg's column rename is built on top of field id, I don't see any Delta/Hudi equivalence are included in the converted metadata
for Delta: "delta.columnMapping.mode":"id","delta.columnMapping.maxColumnId":"4" is missing in the delta log -- see https://docs.databricks.com/en/delta/delta-column-mapping.html for the implementation of 'delta.columnMapping.mode' = 'id'
for Hudi : hudi commit log doesn’t have id, max_column_id populated (fields in https://github.com/apache/hudi/pull/4910/files )
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Spark SQL reproduction in the issue: create an Iceberg table, rename a column, convert it to Hudi and Delta, and inspect the resulting schema and metadata. Read Delta's column-mapping documentation and the referenced Hudi change; done means the converted tables preserve the renamed column and the required field-ID metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100