apache / apache/hudi

[Safe Schema Evolution] Register schema in HMS pre-write to prevent Hudi/HMS schema divergence

Open
#18,008 1 comment 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**Why this task is needed:**
Currently, Hudi performs HMS schema sync as a post-commit operation. This creates a critical failure scenario: if a writer successfully commits data with an evolved schema but the subsequent HMS sync fails, the Hudi table schema and HMS schema diverge. This divergence causes query failures for downstream consumers (Spark, Presto) that rely on HMS for schema metadata, and requires manual intervention to reconcile the schemas (i.e. rollback the commits which introduced schema changes).

HMS schema sync failures can occur for various reasons, one of which is an integration designed to enforce formal schema reviews. For critical datasets, table owners must obtain approvals before implementing schema updates. In these scenarios, direct ALTER TABLE commands are expected to fail, as schema changes are managed via a different system that facilitates the formal review process.

The goal for datasets with strict schema evolution guidelines is to prevent unintended graceful schema evolution by Hudi, which could otherwise lead to schema divergence and impact downstream consumers.

**What needs to be done:**
To prevent this issue, Deltastreamer and Datasource writers should perform HMS schema sync before creating a commit when schema changes are detected and hoodie.datasource.hive_sync.enable=true. If the pre-commit HMS sync fails, the write operation should fail without creating a commit, ensuring that the Hudi table schema and HMS schema always remain consistent. This approach provides fail-fast behavior and eliminates the schema divergence window entirely.

### Task Type

Code improvement/refactoring

### Related Issues

**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.

Contributor guide

No contributing guide indexed for this repository

Research direction

Trace the Deltastreamer and Datasource writer commit paths and their existing HMS schema-sync handling. Check how schema changes are detected and how sync failures affect commit creation; done means both writers fail before creating a commit when schema sync is enabled and fails, with regression coverage for that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.