Increase Resilience of JDBC Sink Connector w/ single, multi-partition topic
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
## Feature request or enhancement
_For feature requests or enhancements, please provide the following information:_
**Which use case/requirement will be addressed by the proposed feature?**
When a JDBC Sink Connector is configured with multiple tasks consuming from a topic with multiple partitions, and schema evolution is enabled, a race condition can occur during DDL execution. If multiple tasks simultaneously detect that a destination table is missing columns, each task independently generates and attempts to execute the same ALTER TABLE statement. The first task to execute succeeds; the remaining tasks fail with a column already exists error and terminate, rather than continuing to process records.
Example error:
`Caused by: org.postgresql.util.PSQLException: ERROR: column "order_status" of relation "customer_orders" already exists`
---
**Implementation ideas (optional)**
When a task encounters a DDL failure because the target column already exists (i.e., it lost the race), rather than throwing a fatal error and terminating the task, the connector should treat the schema as already up-to-date and requeue the consistency check for the affected event, then continue processing. If full idempotency is not feasible, at minimum the connector should retry the schema consistency check Z times before failing.
Contributor guide
Assessment
This issue has not been assessed yet.