[SUPPORT] Hudi not evolving Hive decimal to higher precision and scale
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
An older version of a table schema had a column `xyz` typed as `DECIMAL(10,3)` while a newer data file has the same column typed to `DECIMAL(28,6)`.
```
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Could not convert field Type from DECIMAL(10,3) to DECIMAL(28,6) for field xyz
at org.apache.hudi.hive.util.HiveSchemaUtil.getSchemaDifference(HiveSchemaUtil.java:103)
at org.apache.hudi.hive.HiveSyncTool.syncSchema(HiveSyncTool.java:249)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:184)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:129)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:115)
... 48 more
```
I would expect this to work because values in the field with type `DECIMAL(10,3)` will fit into the field of type `DECIMAL(28,6)`. I think this change would be considered 'backward compatible' and Hudi should just update the schema in Hive to set the column type to `DECIMAL(28,6)`, which would fit the old values as well as new values.
Shouldn't the schema evolve if the new data type is 'bigger' than the previous data type?
I'm running this a Glue 3 job.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.