apache / apache/hudi

[to be discussed] Support having other concurrent writer service execute metadata table compaction plans (outside of ingestion)

Open
#17,908 3 comments 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**
- Implement https://issues.apache.org/jira/browse/HUDI-9407 so that users can specify using `OPTIMISTIC_CONCURRENCY_CONTROL` when creating metadata table writer `org.apache.hudi.metadata.HoodieMetadataWriteUtils#createMetadataWriteConfig`

Add a new config that lists the table services supported for inline scheduling but async execution in metadata table. Initially we can choose to support just compaction and logcompaction. If enabled,
- metadata table writer should still schedule all table service plans inline but not execute the specified types of table service plans inline. Note that this means it should also not re-try these plans in `org.apache.hudi.metadata.HoodieBackedTableMetadataWriter#runPendingTableServicesOperationsAndRefreshTimeline`
- Another concurrent writer can initialize a metadata table writer (passing in OCC as concurrency type in the metadata write config) and execute these plans.

**Why this task is needed:**
For datasets with large metadata table partitions (like RECORD_INDEX) we cannot have metadata table compaction be executed during the write, as that will impact runtimes. Rather, we only schedule the plan inline, and have a separate platform that executes these plans. We need the above features to ensure that we can have specific table services on the metadata table be guaranteed to not execute inline during write. And instead, outside concurrent writers can safely execute these plans. This "async" execution will only take the table lock when transitioning the instants, in order to not block the writer job. We can upstream our implementations for above once we reach consensus.

### Task Type

Code improvement/refactoring

### Related Issues

**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with HUDI-9407 and the metadata writer configuration entry point, org.apache.hudi.metadata.HoodieMetadataWriteUtils#createMetadataWriteConfig. Then trace org.apache.hudi.metadata.HoodieBackedTableMetadataWriter#runPendingTableServicesOperationsAndRefreshTimeline; done means the agreed configuration can schedule compaction and logcompaction without executing or retrying them inline, while an OCC writer can execute them separately.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.