[Safe Schema Evolution] Register schema in HMS pre-write to prevent Hudi/HMS schema divergence
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Task Description
**Why this task is needed:**
Currently, Hudi performs HMS schema sync as a post-commit operation. This creates a critical failure scenario: if a writer successfully commits data with an evolved schema but the subsequent HMS sync fails, the Hudi table schema and HMS schema diverge. This divergence causes query failures for downstream consumers (Spark, Presto) that rely on HMS for schema metadata, and requires manual intervention to reconcile the schemas (i.e. rollback the commits which introduced schema changes).
HMS schema sync failures can occur for various reasons, one of which is an integration designed to enforce formal schema reviews. For critical datasets, table owners must obtain approvals before implementing schema updates. In these scenarios, direct ALTER TABLE commands are expected to fail, as schema changes are managed via a different system that facilitates the formal review process.
The goal for datasets with strict schema evolution guidelines is to prevent unintended graceful schema evolution by Hudi, which could otherwise lead to schema divergence and impact downstream consumers.
**What needs to be done:**
To prevent this issue, Deltastreamer and Datasource writers should perform HMS schema sync before creating a commit when schema changes are detected and hoodie.datasource.hive_sync.enable=true. If the pre-commit HMS sync fails, the write operation should fail without creating a commit, ensuring that the Hudi table schema and HMS schema always remain consistent. This approach provides fail-fast behavior and eliminates the schema divergence window entirely.
### Task Type
Code improvement/refactoring
### Related Issues
**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.
Contributor guide
No contributing guide indexed for this repository
Research direction
Trace the Deltastreamer and Datasource writer commit paths and their existing HMS schema-sync handling. Check how schema changes are detected and how sync failures affect commit creation; done means both writers fail before creating a commit when schema sync is enabled and fails, with regression coverage for that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100