apache / apache/hudi

About full schema evolution, alter column type do not support nest column but can alter inside struct type

Open
#15,858 0 comments 0 reactions 0 assignees View on GitHub
area:schema area:sql from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

reproduce steps:
* create the test table:

{code:java}
CREATE TABLE default.schema_evolution_test (
                          id bigint,
                          test_array array,
                          test_array_struct ARRAY>)
                        USING hudi
                        TBLPROPERTIES (
                          type = 'cow',
                          primaryKey = 'id')
                        LOCATION 's3://trino-ci-test/schema_evolution_test' {code}
* enable the full schema evolution

{code:java}
SET hoodie.schema.on.read.enable=true{code}
* alter the nested table column type, there is exception

{code:java}
SQL A: alter table default.schema_evolution_test alter column test_array_struct type ARRAY>;{code}
{code:java}
Exception: Failed in [alter table default.schema_evolution_test alter column test_array_struct type ARRAY>]
java.lang.IllegalArgumentException: only support update primitive type but find nest column: test_array_struct
        at org.apache.hudi.internal.schema.action.TableChanges$ColumnUpdateChange.updateColumnType(TableChanges.java:89)
        at org.apache.spark.sql.hudi.command.AlterTableCommand.$anonfun$applyUpdateAction$1(AlterTableCommand.scala:149)
  {code}
* alter the inner struct type, it can execute successfully

{code:java}
SQL B: alter table default.schema_evolution_test alter column test_array_struct.element.age type FLOAT;{code}
* Describe the table schema;

{code:java}
desc default.schema_evolution_test;                              
id                      bigint                                                            
test_array              array                                  
test_array_struct       array>    {code}
* Questions:
## is this a bug of alter table column type for Hudi?
## Both SQL A and SQL B is used to change the type from array> to array>, why SQL A do not supported?
## SQL B is worked as expected or not? and SQL A will be supported in next release or future?

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5971
- Type: Bug
- Affects version(s):
- 0.12.2
- 0.13.0

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the table setup and SQL A/SQL B steps, then inspect org.apache.hudi.internal.schema.action.TableChanges.java at line 89 and AlterTableCommand.scala at line 149. Compare the nested-column and whole-column paths; done means the supported behavior is established and the resulting schema matches the intended array> evolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.