apache / apache/paimon

[Improve] Add a listener mechanism for Paimon's commit

Open
#959 11 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

At present, Paimon's commit behavior is not perceived by the outside world, but in some scenarios, especially platform-based scenarios, we may need to monitor Paimon's commit behavior to decide when to perform a compaction.

You may ask why not use Paimon's automatic compaction? When we used Flink for streaming computing, `RocksDBStateBackend` would also perform compaction periodically, which would cause periodic spikes in CPU usage, which is very unfriendly to streaming computing. The behavior of compaction will compete with computing tasks for CPU, and usually the CPU usage of compaction is high, but the duration is short, resulting in low resource utilization.

Therefore, separating the compaction behavior from the streaming job and letting an external platform take charge of the compaction will improve resource utilization and be more friendly to streaming computing.

### Solution

Add a listener for Paimon's commit behavior. This listener supports user-defined extensions, such as reporting to external services through the rest api.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are named in the issue. Start by locating Paimon's commit path and existing extension mechanisms, then determine how a listener can observe commits without coupling compaction to the streaming job. Done means user-defined listeners can receive commit events and support reporting to an external service such as a REST API.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.