Add API to expose Debezium engine and connector activity [DBZ-5432]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 9
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-5432](https://issues.redhat.com/browse/DBZ-5432)
h1. Feature request or enhancement
h2. Which use case/requirement will be addressed by the proposed feature?
Currently users can learn about what Debezium and the connector are doing through the logs. It would be handy, however, if such information can be exposed through an API.
The context for this feature request is that I work for [Airbyte|https://github.com/airbytehq/airbyte]. We embed Debezium in our application and use it in a non-conventional way. Usually Debezium is always on. But we have turn it on and off periodically. Consequently, it is hard to know programmatically what Debezium is doing and when is the best time to turn it off. For example, when no data is coming from the Debezium engine, there are multiple possibilities: 1) there is no change in the database; 2) Debezium is running snapshot for new tables; 3) Debezium is going through change logs that have already been processed. It is proper to shut down the engine for the first scenario, but not for the other two. If an API can tell us what Debezium and the connector is doing, that would be very helpful for our use case and other use cases like monitoring and debugging programmatically.
I have searched the backlog and did not find any existing issue coving this. But it is possible that I was not using the best search term. Please let me know if this is a duplication.
Please also let me know if such an API makes sense. We can work on the implementation if that's the case.
h2. Implementation ideas (optional)
Add a {{DebeziumActivity}} class with two fields:
{code:java}
public enum DebeziumState {
UNSTARTED,
STARTING,
SNAPSHOTTING,
PROCESSING_RECORDS,
WAITING_FOR_RECORDS,
STOPPED
}
public class DebeziumActivity {
private final DebeziumState state;
private final Optional message;
}{code}
The state enum represents what Debezium and the connector is doing at the moment. The optional message provides extra information if there is any.
In [DebeziumEngine|https://github.com/debezium/debezium/blob/main/debezium-api/src/main/java/io/debezium/engine/DebeziumEngine.java], add a new {{StateReporter}} interface:
{code:java}
public static interface StateReporter {
default Optional hasNewState() {
return Optional.empty();
}
} {code}
By default, it returns an empty state, meaning that this report is not implemented.
Users can register a reporting handler when creating the engine. Each connector needs report their status accordingly, which can be returned to the user from this interface in the engine.
Contributor guide
Research direction
Start with debezium-api/src/main/java/io/debezium/engine/DebeziumEngine.java and the proposed DebeziumActivity and DebeziumState types in the issue. Trace how an engine is created and how connectors report status, then determine the API contract and lifecycle states needed. Done means users can obtain the engine and connector activity programmatically, including the listed states and optional message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend-api-design, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100