Add a boolean parameter to decide whether the partition is cascade or not when hive table columns changes
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, when using the HudiHiveSync tool to do hive table columns changes, this happens at HMSDDLExecutor.updateTableDefinition(), this cascade is only decided by META_SYNC_PARTITION_FIELDS:
{code:java}
boolean cascade = syncConfig.getSplitStrings(META_SYNC_PARTITION_FIELDS).size() > 0;{code}
but some scenarios do not need to update partition columns and this will cost a lot of time even the hive might hang when the partition number is large.
Therefore, I want to add a supplement boolean config parameter:
*HIVE_SYNC_PARTITION_CASCADE_WITH_COLUMN_CHANGE*
the default is true, if the users don't want to update the partition then set it to false
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4396
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at HMSDDLExecutor.updateTableDefinition() and trace how META_SYNC_PARTITION_FIELDS is read and how the Hive sync configuration is defined. Add HIVE_SYNC_PARTITION_CASCADE_WITH_COLUMN_CHANGE with a default of true; when false, column changes should not update partition columns or incur the associated partition work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100