Debezium MSK Connector Running but Suddenly Stalling Data Capture [DBZ-8485]

Migrated from [DBZ-8485](https://issues.redhat.com/browse/DBZ-8485)

In order to make your issue reports as actionable as possible, please provide the following information, depending on the issue type.
h1. Bug report

For bug reports, provide this information, please:
h2. What Debezium connector do you use and what version?

We are using the *Debezium MSK Connector* with version 2.7.0 
h2. What is the connector configuration?

```

connector.class=io.debezium.connector.mysql.MySqlConnector 

database.user= 

database.password= 

database.hostname= 

database.port= 

database.server.id= 

database.server.name= 

database.include.list= 

tasks.max=1 

snapshot.mode=schema_only 

key.converter=org.apache.kafka.connect.json.JsonConverter 

key.converter.schemas.enable=false 

value.converter=org.apache.kafka.connect.json.JsonConverter 

value.converter.schemas.enable=false 

config.storage.topic= 

offset.storage.topic= 

offset.storage.replication.factor=2 

topic.creation.default.cleanup.policy=delete 

topic.creation.default.partitions=2 

topic.creation.default.replication.factor=2 

decimal.handling.mode=string 
```
h2. What is the captured database version and mode of deployment?

The database is *RDS MySQL version 8.0.35* (please replace with the version you’re using) deployed in *AWS MSK* using Debezium as the Kafka connector.
h2. What behavior do you expect?

The connector should continuously capture data changes from the source database and write them to the Kafka topic without interruptions.
h2. What behavior do you see?

The connector process remains running, but {*}data capture suddenly stalls{*}, and no new data is published to the Kafka topic.

•The stalls occur unpredictably without prior warning, even though the connector logs indicate it is actively running.

•We observed *gradual memory utilization increases* over time (step-like pattern), but this increase does not directly correlate with the sudden stalls.

•Restarting the connector temporarily resolves the issue, but it recurs after about 30 days.

 

*bytes in per sec for offset topic on datadog metic*

!image-2024-12-04-13-48-48-521.png|width=525,height=200!

*Logs in a normal state*

!image-2024-12-04-13-56-27-578.png|width=724,height=125!

*Logs in an abnormal state*

*!image-2024-12-04-13-58-31-536.png|width=762,height=132!*

*Gradual memory increase metrics*

*!image-2024-12-04-14-03-39-821.png|width=730,height=77!*

 

*Sudden connector restart (appears to be memory-related, but no OOM message observed)*

*!image-2024-12-04-14-05-01-730.png|width=765,height=171!*
h2. Do you see the same behaviour using the latest released Debezium version?

Yes, the issue persists in the latest stable release. Testing with the latest Alpha/Beta/CR version is pending.
h2. Do you have the connector logs, ideally from start till finish?

(You might be asked later to provide [DEBUG/TRACE|https://debezium.io/documentation/reference/stable/operations/logging.html] level log)

 

Yes, the logs are available. Notable observations include:

• Normal operation logs show Committing offsets → flushing outstanding messages → Finished commitOffsets successfully.

• At problematic times, logs alternate between Committing offsets → flushing 0 outstanding messages, with no Finished commitOffsets successfully.
h2. How to reproduce the issue using our [tutorial|https://github.com/debezium/debezium-examples/tree/main/tutorial] deployment?

1.Deploy the Debezium MSK Connector with the provided configuration.

2.Configure it to capture binlogs from a MySQL database with moderate to high traffic.

3.Observe memory utilization over time in *CloudWatch* or equivalent monitoring tools.

4.The memory utilization will display a step-like increase pattern after a few hours of operation.
h1. Feature request or enhancement

For feature requests or enhancements, provide this information, please:
h2. Which use case/requirement will be addressed by the proposed feature?

h2. Implementation ideas (optional)

Start with the supplied connector configuration, offset-commit log sequences, and CloudWatch memory metrics. Reproduce the behavior with the Debezium tutorial deployment and MySQL traffic, then use the normal and abnormal logs to define a fix that keeps capture publishing continuously without requiring a restart.

Written by the indexing model from the issue text.