apache / apache/hudi

[SUPPORT] Hudi not evolving Hive decimal to higher precision and scale

Open
#8,196 2 comments 0 reactions 1 assignee Claimed by @danny0405 View on GitHub
area:aws area:schema priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

An older version of a table schema had a column `xyz` typed as `DECIMAL(10,3)` while a newer data file has the same column typed to `DECIMAL(28,6)`.

```
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Could not convert field Type from DECIMAL(10,3) to DECIMAL(28,6) for field xyz
at org.apache.hudi.hive.util.HiveSchemaUtil.getSchemaDifference(HiveSchemaUtil.java:103)
at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:249)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:184)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:129)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:115)
... 48 more
```

I would expect this to work because values in the field with type `DECIMAL(10,3)` will fit into the field of type `DECIMAL(28,6)`. I think this change would be considered 'backward compatible' and Hudi should just update the schema in Hive to set the column type to `DECIMAL(28,6)`, which would fit the old values as well as new values.

Shouldn't the schema evolve if the new data type is 'bigger' than the previous data type?

I'm running this a Glue 3 job.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.