apache / apache/hudi

Add separate control for compaction operation sync/async mode

Open
#15,256 3 comments 0 reactions 1 assignee Claimed by @cshuo View on GitHub
area:table-service component:compaction engine:flink priority:low status:pr-available type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

*Problem Review*
The compact operation sync/async in CompactionFunction is now controlled by FlinkOptions#COMPACTION_ASYNC_ENABLED
{code:java}
public CompactFunction(Configuration conf) {
this.conf = conf;
this.asyncCompaction = StreamerUtil.needsAsyncCompaction(conf);
}
{code}
While in fact it cannot be switched to sync mode because the pipeline defined by sync compaction will only include the clean but not compact operators.
{code:java}
// compaction
if (StreamerUtil.needsAsyncCompaction(conf)) {
return Pipelines.compact(conf, pipeline);
} else {
return Pipelines.clean(conf, pipeline);
}
{code}
*Improvement*
Add another separate control switch for compaction operation sync/async mode.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4329
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.