apache / apache/pulsar-connectors

[improve][io] Migrate Debezium from the 3.4.x line to 3.6.0.Final

Open
#63 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
26
Forks
25
Avg merge
17h 7m
Merged PRs (30d)
1

Description

## Motivation

The repository pins `debezium = "3.4.2.Final"` (released 2026-03-02). The current Debezium release is **3.6.0.Final** (2026-07-01), so we are two minor versions behind.

Debezium has no LTS line — fixes land on the newest minor only. Once #64 bumps us to `3.4.3.Final` (2026-03-30, the last patch the 3.4 line will ever receive), the 3.4 branch is end-of-line and we stop receiving bug and security fixes entirely. Each additional minor release makes the eventual migration more expensive.

## Scope

Bump `debezium` in `gradle/libs.versions.toml` from `3.4.3.Final` to `3.6.0.Final` and fix any resulting breakage.

**The connector set is unchanged.** Comparing the `debezium-bom` contents for 3.4.2 and 3.6.0, the only addition is `debezium-connector-common` (an internal refactor). No wrapper module gains or loses a target.

## Risks

The real risk is our coupling to Debezium **internals**, which are not API-stable across minor versions:

- `debezium/core` implements the schema-history SPI (`PulsarSchemaHistory`) against `io.debezium.storage` / `SchemaHistory` interfaces.
- `kafka-connect-adaptor` (`AbstractKafkaConnectSource`) starts Debezium tasks directly and depends on connector/task config-key conventions.
- `DebeziumSource` subclasses assert on Debezium's `connector.class` / `task.class` values.

Issue #61 is a concrete example of this coupling already breaking against 3.4.x (multi-partition SQL Server connectors never receive `task.id` because we bypass `taskConfigs()`). Expect similar sharp edges on a minor bump; #61 should ideally be fixed first, or at least understood, since it touches the same task-startup path.

## Prerequisites

The integration tests added in #58 (MongoDB), #60 (SQL Server), and #62 (Oracle) should be **merged before this migration**. Prior to those, a Debezium bump was validated only by the MySQL and Postgres tests. With all five source connectors exercised against real databases via Testcontainers, this upgrade either passes CI across the board or points directly at the connector that broke.

## Note: this will not fix the Testcontainers conflict

`debezium-bom:3.4.2.Final` pins Testcontainers core to `2.0.2`; `3.6.0.Final` (via `debezium-build-parent`) pins `2.0.3`. Both are 2.x, while the specialized Testcontainers modules (e.g. `org.testcontainers:oracle-free`) still target 1.21.x and reference shaded classes removed in 2.x. Debezium-module integration tests must keep using the `GenericContainer` pattern (as `debezium/mysql` and `debezium/oracle` do) regardless of this upgrade — see the discussion in #62.

## Suggested order

1. Merge #58, #60, #62 (integration-test safety net)
2. Land the `3.4.3.Final` patch bump (low risk, free fixes)
3. Address or characterize #61
4. Migrate to `3.6.0.Final` and let the five connector integration tests validate it

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in gradle/libs.versions.toml and inspect the Debezium version and BOM changes. Read PulsarSchemaHistory, AbstractKafkaConnectSource, and DebeziumSource, then run the MongoDB, SQL Server, Oracle, MySQL, and Postgres integration tests from #58, #60, and #62. Done means the migration reaches 3.6.0.Final, the connector set remains unchanged, and all five integrations pass without regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, build-system, testing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.