apache / apache/hudi

Add a boolean parameter to decide whether the partition is cascade or not when hive table columns changes

Open
#15,280 1 comment 0 reactions 0 assignees View on GitHub
component:catalog-sync engine:hive from-jira priority:critical status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently, when using the HudiHiveSync tool to do hive table columns changes, this happens at HMSDDLExecutor.updateTableDefinition(), this cascade is only decided by META_SYNC_PARTITION_FIELDS:
{code:java}
boolean cascade = syncConfig.getSplitStrings(META_SYNC_PARTITION_FIELDS).size() > 0;{code}
but some scenarios do not need to update partition columns and this will cost a lot of time even the hive might hang when the partition number is large. 

Therefore, I want to add a supplement boolean config parameter: 

*HIVE_SYNC_PARTITION_CASCADE_WITH_COLUMN_CHANGE*

the default is true, if the users don't want to update the partition then set it to false

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4396
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at HMSDDLExecutor.updateTableDefinition() and trace how META_SYNC_PARTITION_FIELDS is read and how the Hive sync configuration is defined. Add HIVE_SYNC_PARTITION_CASCADE_WITH_COLUMN_CHANGE with a default of true; when false, column changes should not update partition columns or incur the associated partition work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.