apache / apache/hudi

Users should be able to dictate which partition in MDT needs to be built async and which one is inline

Open
#15,950 1 comment 0 reactions 0 assignees View on GitHub
area:metadata-table from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

In metadata writer code, if async indexing is enabled, we assume every partition needs to be built out async. but users can choose to pick 2 partitions for inline and 2 for async. As of now, we don't have that flexibility. 

 

Code snippet from HoodieBackedTableMetadataWriter
{code:java}
// if metadata table exists, then check if any of the enabled partition types needs to be initialized
// NOTE: It needs to be guarded by async index config because if that is enabled then initialization happens through the index scheduler.
if (!dataWriteConfig.isMetadataAsyncIndex()) {
Set inflightAndCompletedPartitions = getInflightAndCompletedMetadataPartitions(dataMetaClient.getTableConfig());
LOG.info("Async metadata indexing enabled and following partitions already initialized: " + inflightAndCompletedPartitions);
List partitionsToInit = this.enabledPartitionTypes.stream()
.filter(p -> !inflightAndCompletedPartitions.contains(p.getPartitionPath()) && !MetadataPartitionType.FILES.equals(p))
.collect(Collectors.toList());
// if there are no partitions to initialize or there is a pending operation, then don't initialize in this round
if (partitionsToInit.isEmpty() || anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return;
}

String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
initTableMetadata(); // re-init certain flags in BaseTableMetadata
initializeEnabledFileGroups(dataMetaClient, createInstantTime, partitionsToInit);
initialCommit(createInstantTime, partitionsToInit);
updateInitializedPartitionsInTableConfig(partitionsToInit);
} {code}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6205
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in HoodieBackedTableMetadataWriter at the metadata-partition initialization logic shown in the issue, then trace the async metadata indexing configuration and partition scheduling path. Define how users select partitions for inline versus async construction, and verify that initialization respects those selections rather than treating all partitions uniformly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.