Event data was too large to fit in the batch [DBZ-3945]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 9
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-3945](https://issues.redhat.com/browse/DBZ-3945)
I have a problem with loading data with SQL Server Connector to Azure Event Hub. After some data are successfully exported I get error shown below that Event data was too large to fit in the batch.
{code:java}
INFO [io.deb.ser.ConnectorLifecycle] (pool-7-thread-1) Connector completed: success = 'false', message = 'Stopping connector after error in the application's handler method: io.debezium.DebeziumException: Event data was too large to fit in the batch', error = '{}': io.debezium.DebeziumException: io.debezium.DebeziumException: Event data was too large to fit in the batch
at io.debezium.server.eventhubs.EventHubsChangeConsumer.handleBatch(EventHubsChangeConsumer.java:159)
at io.debezium.embedded.ConvertingEngineBuilder.lambda$notifying$2(ConvertingEngineBuilder.java:83)
at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:821)
at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:188)
at io.debezium.server.DebeziumServer.lambda$start$1(DebeziumServer.java:145)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.debezium.DebeziumException: Event data was too large to fit in the batch
at io.debezium.server.eventhubs.EventHubsChangeConsumer.handleBatch(EventHubsChangeConsumer.java:146)
... 7 more
{code}
Configuration used during this execution is shown below.
{code:java}
debezium.sink.type=eventhubs
debezium.sink.kafka.producer.key.serializer=org.apache.kafka.common.serializati>
debezium.sink.kafka.producer.value.serializer=org.apache.kafka.common.serializa>
debezium.sink.eventhubs.connectionstring=Endpoint=
debezium.sink.eventhubs.hubname=heo
debezium.sink.eventhubs.partitionid=0
debezium.sink.eventhubs.max.batch.size=1048576
debezium.source.database.history.kafka.pool.interval.ms=300
debezium.source.database.history.snapshot.fetch.size=20000
debezium.source.connector.class=io.debezium.connector.sqlserver.SqlServerConnec>
debezium.source.database.hostname=
debezium.source.database.port=
debezium.source.database.user=
debezium.source.database.password=
debezium.source.database.dbname=
debezium.source.table.include.list=<109 tables here>
debezium.source.database.server.name=
debezium.source.database.history=io.debezium.relational.history.MemoryDatabaseH>
debezium.source.database.history.
debezium.source.database.history.kafka.topic=
debezium.source.offset.storage.file.filename=/tmp/offsets.dat
debezium.source.offset.flush.interval.ms=500
debezium.source.database.history.key.converter.schemas.enable=false
debezium.source.database.history.value.converter.schemas.enable=false
debezium.source.database.history.consumer.security.protocol=SASL_SSL
debezium.source.database.history.consumer.sasl.mechanism=PLAIN
debezium.source.database.history.consumer.sasl.jaas.config=
{code}
The size is set according to the Event Hub message limits.
Could someone please point how to find the reason of this issue?
Contributor guide
Research direction
Start at debezium/server/eventhubs/EventHubsChangeConsumer.java, especially handleBatch at lines 146 and 159, and review how debezium.sink.eventhubs.max.batch.size is applied against Azure Event Hubs limits. Reproduce the failure with the supplied configuration and determine why a batch is rejected; done means the cause and a validated fix or configuration guidance are documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, java, sql
- Domain
- backend, cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100