debezium / debezium/dbz

More logging when skipping events [DBZ-7719]

Open
#970 0 comments 0 reactions 1 assignee Claimed by @vjuranek View on GitHub
component/postgresql-connector good first issue migrated-from-jira type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-7719](https://issues.redhat.com/browse/DBZ-7719)

h2. Which use case/requirement will be addressed by the proposed feature?
When using Postgres connector and Debezium has to process a larger WAL, it could happen that the application looks like stuck because it skips a lot of messages, and so it doesn't call the {{handleBatch}} method and also doesn't print any log message.
For example, if the WAL is sorted and the events from the watched tables are at the end.
In this situation, it would be useful to see INFO or WARN messages that the application is running, but skipping a lot of events.

h2. Implementation ideas (optional)
Actually, there is code that is very close to this: https://github.com/debezium/debezium/blob/main/debezium-connector-postgres/src/main/java/io/debezium/connector/postgresql/PostgresStreamingChangeEventSource.java#L401
Its concern is WAL size, but it looks like it doesn't count with all the event types - in our case, it reported 10000 skipped events after 50 minutes. When processing events (not skipping), the machine was able to process 20k events per second, so this doesn't look like an performance issue.
So if this code could be improved to count with all events types, and {{GROWING_WAL_WARNING_LOG_INTERVAL}} constant to be configurable, it would be perfect.

What is similar is the {{logStatistics}} method in [BaseSourceTask|https://github.com/debezium/debezium/blob/main/debezium-core/src/main/java/io/debezium/connector/common/BaseSourceTask.java#L302] but it is IMHO too high-level and doesn't know about skipped events.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.