Filtered heartbeat events cause unbounded WAL growth [DBZ-9490]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-9490](https://issues.redhat.com/browse/DBZ-9490)
As discussed on the mailing list, a user was upgrading from Debezium 2.5 to Debezium 3.1 and was noticing that there was some behavior differences with offset management when a transformation filters an event (in their case heartbeats).
{quote}
When a `SourceRecord` goes through the transformation chain [1], the `producerRecord` is `null` and therefore `AbstractWorkerSourceTask` calls `recordDropped`. This ultimately calls `SourceTask#commitRecord`, which for Debezium is a no-op. When the `producerRecord` is not `null`, the `prepareToSendRecord` [2] method is called. This eventually calls `SourceTask#commit`, and this is where Debezium synchronizes the offset state with the replication slot.
So if you have a period where all you have are heartbeat events and they're being filtered and not sent to Kafka, then the replication slot's confirmed LSN won't be updated, which explains your WAL growth. The reason you observe a difference in behavior appears to be related to a change in DBZ-7816 [3].
For now, as a workaround, simply don't filter the heartbeat event and let it go the Kafka topic.
[1]: https://github.com/apache/kafka/blob/d6133f6997e0de3931f3d98b30950aeeefd23e70/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L415-L416
[2]: https://github.com/apache/kafka/blob/d6133f6997e0de3931f3d98b30950aeeefd23e70/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L425
[3]: https://issues.redhat.com/browse/DBZ-7816
{quote}
In DBZ-7816 the logic was changed so that we only rely on committed offsets to the offset topic and the logic in {{commitRecord}} uses the same logic, which is the crux of why a filtered event, which only calls {{commitRecord}} and never updates the Kafka offset topic, has this issue on newer builds of Debezium.
The present workaround is to simply not filter heartbeat events, but that is not desired or ideal in certain environments.
Contributor guide
Assessment
This issue has not been assessed yet.