apache / apache/paimon

[Bug] compact_database StreamingCompactorSource Busy(max) 100%

Open
#5,463 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

1.0

### Compute Engine

flink1.17.1

### Minimal reproduce step

public class CompactDatabaseTestSourceBusy extends CompactActionITCaseBase {
public static void main(String[] args) throws Exception {
CompactDatabaseAction action;
action =
createAction(
CompactDatabaseAction.class,
"compact_database",
"--warehouse",
"hdfs:///user/paimon/warehouse_dw",
"--mode",
"combined",
"--including_databases",
"dap_dev_test",
"--table_conf",
"snapshot.num-retained.min=3",
"--table_conf",
"snapshot.time-retained=5m",
"--table_conf",
"full-compaction.delta-commits=5",
"--table_conf",
CoreOptions.CONTINUOUS_DISCOVERY_INTERVAL.key() + "=120s");

Configuration conf = new Configuration();
conf.setString(RestOptions.BIND_PORT, "8081-8089");
conf.setBoolean("rest.flamegraph.enabled",true);
StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(conf);
env.enableCheckpointing(30000);
action.withStreamExecutionEnvironment(env).build();
env.executeAsync();

}
}

![Image](https://github.com/user-attachments/assets/adbcbe51-8373-4b6d-b3a8-d6a3f376ca5f)

### What doesn't meet your expectations?

My question is why the following two operators have consistently been showing "Busy(max): 100%":
Source: Combine-MultiBucketTables--StreamingCompactorSource
Source: Combined-UnawareBucketTables-StreamingCompactorSource

Under this warehouse, there are only 8 databases, and the dap_dev_test database contains only 2 tables. The computation for merging tables shouldn't consume this much CPU. Moreover, through analyzing the source code, I found that the Thread.sleep(monitorInterval) in the following two functions is indeed working properly:

org.apache.paimon.flink.source.operator.CombinedAwareStreamingSource.Reader#pollNext

org.apache.paimon.flink.source.operator.CombinedUnawareStreamingSource.Reader#pollNext

This indicates that the program enters these functions and sleeps for 120 seconds as intended. Therefore, it is unclear what operations are causing the two operators to remain in the "Busy(max): 100%" state.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the CompactDatabaseTestSourceBusy reproduction and inspect CombinedAwareStreamingSource.Reader#pollNext and CombinedUnawareStreamingSource.Reader#pollNext. Run the compact_database job with the supplied Flink 1.17.1 configuration, then trace what keeps both operators busy despite the 120-second sleep. Done means the Busy(max) behavior is explained and the appropriate fix or verification is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.