debezium / debezium/dbz

Increase Resilience of JDBC Sink Connector w/ single, multi-partition topic

Open
#2,302 0 comments 0 reactions 0 assignees View on GitHub
component/jdbc-connector type/enhancement
Dominant language
HTML
Stars
6
Forks
8
Avg merge
2d 19h
Merged PRs (30d)
1

Description

## Feature request or enhancement

_For feature requests or enhancements, please provide the following information:_

**Which use case/requirement will be addressed by the proposed feature?**

When a JDBC Sink Connector is configured with multiple tasks consuming from a topic with multiple partitions, and schema evolution is enabled, a race condition can occur during DDL execution. If multiple tasks simultaneously detect that a destination table is missing columns, each task independently generates and attempts to execute the same ALTER TABLE statement. The first task to execute succeeds; the remaining tasks fail with a column already exists error and terminate, rather than continuing to process records.

Example error:

`Caused by: org.postgresql.util.PSQLException: ERROR: column "order_status" of relation "customer_orders" already exists`

---

**Implementation ideas (optional)**

When a task encounters a DDL failure because the target column already exists (i.e., it lost the race), rather than throwing a fatal error and terminating the task, the connector should treat the schema as already up-to-date and requeue the consistency check for the affected event, then continue processing. If full idempotency is not feasible, at minimum the connector should retry the schema consistency check Z times before failing.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.