debezium / debezium/dbz

TimeOutException: Topic not present in metadata after 10000ms [DBZ-8578]

Open
#62 0 comments 0 reactions 0 assignees View on GitHub
component/debezium-server migrated-from-jira type/bug
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

Migrated from [DBZ-8578](https://issues.redhat.com/browse/DBZ-8578)

h1. Bug report

For bug reports, provide this information, please:
h2. What Debezium connector do you use and what version?

debezium server (3.0.2-final) with MySqlConnector

h2. What is the connector configuration?

```

# sink config
debezium.sink.type=kafka
debezium.sink.kafka.producer.bootstrap.servers=XXX
debezium.sink.kafka.producer.key.serializer=org.apache.kafka.common.serialization.StringSerializer
debezium.sink.kafka.producer.value.serializer=org.apache.kafka.common.serialization.StringSerializer
debezium.sink.kafka.producer.security.protocol=SASL_SSL
debezium.sink.kafka.producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='XXX' password='XXX';
debezium.sink.kafka.producer.sasl.mechanism=PLAIN
debezium.sink.kafka.producer.client.dns.lookup=use_all_dns_ips
debezium.sink.kafka.producer.client=PLAIN
debezium.sink.kafka.producer.client.id=vmCDC-SX
debezium.sink.kafka.producer.compression.type=lz4
debezium.sink.kafka.producer.creation.enable=false

# source config

debezium.source.connector.class=io.debezium.connector.mysql.MySqlConnector
debezium.source.database.hostname=vmCDC.local
debezium.source.database.port=3306
debezium.source.database.user=XXX
debezium.source.database.password=XXX
debezium.source.database.server.id=1
debezium.source.topic.prefix=XXX.test.events
debezium.source.database.include.list=db
debezium.source.column.exclude.list=db.tpltsk_issuestate.IconData
debezium.source.table.include.list=db.rel_country,db.rel_municipality,db.rel_company,db.rel_address,db.rel_relation,db.rel_supplieractivity,db.rel_communicationchannel,db.fin_bankaccount,db.rel_supplieractivitytype,db.sec_user,db.rel_communication,db.rel_financial,db.tpltsk_issuestate,db.tpltsk_issue
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.schema.history.internal.kafka.bootstrap.servers=XXX
debezium.source.schema.history.internal.kafka.sasl.mechanism=PLAIN
debezium.source.schema.history.internal.kafka.topic=XXX.test.events-internal
debezium.source.schema.history.internal.kafka.security.protocol=SASL_SSL
debezium.source.schema.history.internal.kafka.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='XXX' password='XXX';
debezium.source.schema.history.internal.consumer.security.protocol=SASL_SSL
debezium.source.schema.history.internal.consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='XXX' password='XXX';
debezium.source.schema.history.internal.consumer.sasl.mechanism=PLAIN
debezium.source.schema.history.internal.producer.security.protocol=SASL_SSL
debezium.source.schema.history.internal.producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='XXX' password='XXX';
debezium.source.schema.history.internal.producer.sasl.mechanism=PLAIN
debezium.source.include.schema.changes=false
debezium.source.topic.creation.default.compression.type=lz4
debezium.source.topic.creation.enable=false
debezium.source.provide.transaction.metadata=true
debezium.source.tombstones.on.delete=false

debezium.source.snapshot.mode=no_data

# signaling => listens to kafka topic test-signal and executes the commands

debezium.source.signal.enabled.channels=kafka
debezium.source.signal.kafka.topic=XXX-signal
debezium.source.signal.kafka.bootstrap.servers=XXX
debezium.source.signal.kafka.sasl.mechanism=PLAIN
debezium.source.signal.kafka.security.protocol=SASL_SSL
debezium.source.signal.kafka.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='XXX' password='XXX';
debezium.source.signal.consumer.security.protocol=SASL_SSL
debezium.source.signal.consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='XXX' password='XXX';
debezium.source.signal.consumer.sasl.mechanism=PLAIN
debezium.source.signal.consumer.client.id=signal-listener-for-kafka-agent
debezium.source.signal.producer.security.protocol=SASL_SSL
debezium.source.signal.producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username='XXX' password='XXX';
debezium.source.signal.producer.sasl.mechanism=PLAIN

# format
debezium.format.key=json
debezium.format.key.schemas.enable=false
debezium.format.value=json
debezium.format.value.schemas.enable=false
debezium.format.header=json

# transformations
debezium.transforms=beginFilter,endFilter,reroute,tenantInsert
debezium.transforms.reroute.type=org.apache.kafka.connect.transforms.RegexRouter
debezium.transforms.reroute.regex=XXX.test.events.([^.]*).(.*)
debezium.transforms.reroute.replacement=XXX.test.events
debezium.transforms.beginFilter.topic.regex=XXX.test.events.transaction
debezium.transforms.beginFilter.type=io.debezium.transforms.Filter
debezium.transforms.beginFilter.language=jsr223.groovy
debezium.transforms.beginFilter.condition=value.status != 'BEGIN'
debezium.transforms.endFilter.topic.regex=XXX.test.events.transaction
debezium.transforms.endFilter.type=io.debezium.transforms.Filter
debezium.transforms.endFilter.language=jsr223.groovy
debezium.transforms.endFilter.condition=!(value.status == 'END' && value.event_count == 0)
debezium.transforms.tenantInsert.type=org.apache.kafka.connect.transforms.InsertField$Value
debezium.transforms.tenantInsert.static.field=tenantId
debezium.transforms.tenantInsert.static.value=XXX

# logging
quarkus.log.console.json=false
quarkus.log.level=INFO
quarkus.http.port=8080

```
h2. What is the captured database version and mode of deployment?

(E.g. on-premises, with a specific cloud provider, etc.)

on premise database : mysql => Version 5.6
h2. What behavior do you expect?

This not to happen
h2. What behavior do you see?

This works fine for 99% but sometimes I get this exception: 
!image-2025-01-15-19-57-07-479.png!
h2. Do you see the same behaviour using the latest released Debezium version?

(Ideally, also verify with latest Alpha/Beta/CR version)

i'm using version 3.0.2 of debezium server and version 3 is latest release

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the intermittent exception with Debezium Server 3.0.2, MySQL 5.6, and the provided Kafka sink and source configuration. Trace the path associated with “Topic not present in metadata” and verify that the configured topics become available reliably without the timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
kafka, mysql
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.