Users should be able to dictate which partition in MDT needs to be built async and which one is inline
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
In metadata writer code, if async indexing is enabled, we assume every partition needs to be built out async. but users can choose to pick 2 partitions for inline and 2 for async. As of now, we don't have that flexibility.
Code snippet from HoodieBackedTableMetadataWriter
{code:java}
// if metadata table exists, then check if any of the enabled partition types needs to be initialized
// NOTE: It needs to be guarded by async index config because if that is enabled then initialization happens through the index scheduler.
if (!dataWriteConfig.isMetadataAsyncIndex()) {
Set inflightAndCompletedPartitions = getInflightAndCompletedMetadataPartitions(dataMetaClient.getTableConfig());
LOG.info("Async metadata indexing enabled and following partitions already initialized: " + inflightAndCompletedPartitions);
List partitionsToInit = this.enabledPartitionTypes.stream()
.filter(p -> !inflightAndCompletedPartitions.contains(p.getPartitionPath()) && !MetadataPartitionType.FILES.equals(p))
.collect(Collectors.toList());
// if there are no partitions to initialize or there is a pending operation, then don't initialize in this round
if (partitionsToInit.isEmpty() || anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return;
}
String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
initTableMetadata(); // re-init certain flags in BaseTableMetadata
initializeEnabledFileGroups(dataMetaClient, createInstantTime, partitionsToInit);
initialCommit(createInstantTime, partitionsToInit);
updateInitializedPartitionsInTableConfig(partitionsToInit);
} {code}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6205
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in HoodieBackedTableMetadataWriter at the metadata-partition initialization logic shown in the issue, then trace the async metadata indexing configuration and partition scheduling path. Define how users select partitions for inline versus async construction, and verify that initialization respects those selections rather than treating all partitions uniformly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100