[SUPPORT][RFC] Support staged / incremental Metadata Table (MDT) initialization for large tables with long partition history
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Problem:
When enabling the Metadata Table (MDT) on existing Hudi tables that have accumulated a large partition history, MDT bootstrap becomes a hard blocker for the writer. In our production setup we routinely operate tables with the following characteristics:
- Daily partitioning strategy (one partition per day)
- ~2 years of retained partitions (≈ 700+ partitions)
- Hundreds of thousands of files per partition
At this scale, the initial HoodieBackedTableMetadataWriter bootstrap performs a full-table FILES (and optionally COLUMN_STATS / PARTITION_STATS / BLOOM_FILTERS) scan across all historical partition before the writer is allowed to resume normal ingestion. The consequences as follows:
1. The upstream writer (Flink / Spark streaming) has to be paused for a very long window while MDT is being populated.
2. Business SLAs on ingestion latency are broken during the one-time MDT rollout, which in practice discourages teams from turning MDT on at all.
### Proposal
We would like to open a discussion on a staged / phased MDT initialization flow, roughly:
1. Phase 1 — Foreground bootstrap of hot partitions only.
During the initial write-side enablement, only bootstrap the partition(s) that the current writer is actively producing into (e.g. today's partition, or the last N partitions defined by a config knob). This should complete in seconds/minutes and unblock the writer.
2. Phase 2 — Asynchronous / offline backfill of historical partitions.
Provide a Spark SQL CALL procedure that can be triggered independently — e.g. `CALL run_metadata_backfill(table => '...', partitions => '...', parallelism => ...)`. This job walks historical partitions and appends the corresponding MDT records
incrementally, without blocking the online writer. It should be resumable, idempotent, and safe to run concurrently with the writer (or at least with well-defined isolation via existing lock providers / TxnManager).
3. Phase 3 — Reader activation gate.
MDT should stay invisible / disabled for readers until an explicit "activation" step is performed (after Phase 2 completes). Concretely, readers should either (a) fall back to filesystem listing for partitions that have not yet been backfilled, or (b) treat MDT as unavailable until
a table-level flag / property (e.g. `hoodie.metadata.ready = true`) is set. This avoids the pitfall of readers getting a partially-populated MDT view during backfill and returning incorrect / incomplete results.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with HoodieBackedTableMetadataWriter and the existing Spark SQL CALL procedure framework. Trace how bootstrap scans partitions and how metadata visibility is controlled, then review the lock providers and TxnManager for concurrent backfill behavior. Done means hot partitions unblock the writer, historical backfill is resumable and idempotent, and readers do not use incomplete metadata before activation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100