Infinite Loop in Outbox Router when topic does not exists [DBZ-3652]
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
Migrated from [DBZ-3652](https://issues.redhat.com/browse/DBZ-3652)
Marked as a bug, but depends on what configuration means.
We are using AWS RDS as a Postgres database based on Postgres 11 version.
We are using *Debezium 1.4.1* version with *pgoutput* and *outbox router.*
Recently, I've been testing what will happen if the topic will not be configured in Kafka.
Normally, in that case, connector login warns & stuck in an infinite loop as the task is unable to produce an event to Kafka:
{code:java}
WARN [Producer clientId=connector-producer-piotrek-test-3.outbox_events_debezium_connector-0] Error while fetching metadata with correlation id 3884 : {piotrek-test-123.partners.feature-flag-events=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
{code}
So, in that case, even when I set up *event.processing.failure.handling.mode* still raises the errors and is stuck in the loop forever.
As an outbox router is configured to be able to fetch data from one table and route them to multiple different topics, one typo can block the entire queue from consuming & bloat databases as the connector will be stuck on a single event forever/until someone creates a topic.
Connector configuration looks
{code:json}
{
"connector.class": "io.debezium.connector.postgresql.PostgresConnector", "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter", "slot.name": "debezium",
"tasks.max": "4",
"publication.name": "outbox_publication",
"transforms": "outbox",
"tombstones.on.delete": "false", "transforms.outbox.route.topic.replacement": "piotrek-test-3.${routedByValue}",
"publication.autocreate.mode": "disabled",
"database.user": "XXXX",
"database.dbname": "XXXX", "transforms.outbox.table.fields.additional.placement": "uuid:envelope:uuid,event_type:envelope:event_type,timestamp:envelope:timestamp",
"transforms.outbox.table.field.event.key": "partition_key", "transforms.outbox.table.field.event.timestamp": "timestamp", "database.server.name": "XXXX",
"transforms.outbox.table.field.event.id": "uuid", "transforms.outbox.route.by.field": "topic_name", "event.processing.failure.handling.mode": "skip",
"database.port": "5432",
"plugin.name": "pgoutput",
"topic.creation.enable": "false",
"database.hostname": "XXXX",
"database.password": "XXXX",
"name": "outbox_events_debezium_connector", "transforms.outbox.table.field.event.payload": "proto_payload", "table.include.list": "public.outbox_events",
"snapshot.mode": "exported", "transforms.outbox.table.field.event.payload.id": "partition_key"
}
{code}
Contributor guide
Research direction
No source files or tests are named in the report. Start by reproducing the missing-topic case with the shown PostgreSQL/Debezium outbox-router configuration and event.processing.failure.handling.mode=skip; done means the connector no longer remains in an infinite loop when Kafka reports UNKNOWN_TOPIC_OR_PARTITION.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, kafka, postgresql
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100