debezium / debezium/dbz

Event data was too large to fit in the batch [DBZ-3945]

Open
#504 0 comments 0 reactions 0 assignees View on GitHub
component/debezium-server migrated-from-jira type/task
Dominant language
HTML
Stars
6
Forks
9
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-3945](https://issues.redhat.com/browse/DBZ-3945)

I have a problem with loading data with SQL Server Connector to Azure Event Hub. After some data are successfully exported I get error shown below that Event data was too large to fit in the batch.

 

 
{code:java}
INFO [io.deb.ser.ConnectorLifecycle] (pool-7-thread-1) Connector completed: success = 'false', message = 'Stopping connector after error in the application's handler method: io.debezium.DebeziumException: Event data was too large to fit in the batch', error = '{}': io.debezium.DebeziumException: io.debezium.DebeziumException: Event data was too large to fit in the batch
at io.debezium.server.eventhubs.EventHubsChangeConsumer.handleBatch(EventHubsChangeConsumer.java:159)
at io.debezium.embedded.ConvertingEngineBuilder.lambda$notifying$2(ConvertingEngineBuilder.java:83)
at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:821)
at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:188)
at io.debezium.server.DebeziumServer.lambda$start$1(DebeziumServer.java:145)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.debezium.DebeziumException: Event data was too large to fit in the batch
at io.debezium.server.eventhubs.EventHubsChangeConsumer.handleBatch(EventHubsChangeConsumer.java:146)
... 7 more
{code}
 

 

Configuration used during this execution is shown below.

 
{code:java}
debezium.sink.type=eventhubs
debezium.sink.kafka.producer.key.serializer=org.apache.kafka.common.serializati>
debezium.sink.kafka.producer.value.serializer=org.apache.kafka.common.serializa>
debezium.sink.eventhubs.connectionstring=Endpoint=
debezium.sink.eventhubs.hubname=heo
debezium.sink.eventhubs.partitionid=0
debezium.sink.eventhubs.max.batch.size=1048576
debezium.source.database.history.kafka.pool.interval.ms=300
debezium.source.database.history.snapshot.fetch.size=20000
debezium.source.connector.class=io.debezium.connector.sqlserver.SqlServerConnec>
debezium.source.database.hostname=
debezium.source.database.port=
debezium.source.database.user=
debezium.source.database.password=
debezium.source.database.dbname=
debezium.source.table.include.list=<109 tables here>
debezium.source.database.server.name=
debezium.source.database.history=io.debezium.relational.history.MemoryDatabaseH>
debezium.source.database.history.
debezium.source.database.history.kafka.topic=
debezium.source.offset.storage.file.filename=/tmp/offsets.dat
debezium.source.offset.flush.interval.ms=500
debezium.source.database.history.key.converter.schemas.enable=false
debezium.source.database.history.value.converter.schemas.enable=false
debezium.source.database.history.consumer.security.protocol=SASL_SSL
debezium.source.database.history.consumer.sasl.mechanism=PLAIN
debezium.source.database.history.consumer.sasl.jaas.config=

{code}
 

The size is set according to the Event Hub message limits. 

Could someone please point how to find the reason of this issue?

Contributor guide

Open the contributing guide

Research direction

Start at debezium/server/eventhubs/EventHubsChangeConsumer.java, especially handleBatch at lines 146 and 159, and review how debezium.sink.eventhubs.max.batch.size is applied against Azure Event Hubs limits. Reproduce the failure with the supplied configuration and determine why a batch is rejected; done means the cause and a validated fix or configuration guidance are documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, java, sql
Domain
backend, cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.