apache / apache/pulsar-connectors
[improve][io] Migrate Debezium from the 3.4.x line to 3.6.0.Final
- Dominant language
- Java
- Stars
- 26
- Forks
- 25
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 1
Description
## Motivation
The repository pins `debezium = "3.4.2.Final"` (released 2026-03-02). The current Debezium release is **3.6.0.Final** (2026-07-01), so we are two minor versions behind.
Debezium has no LTS line — fixes land on the newest minor only. Once #64 bumps us to `3.4.3.Final` (2026-03-30, the last patch the 3.4 line will ever receive), the 3.4 branch is end-of-line and we stop receiving bug and security fixes entirely. Each additional minor release makes the eventual migration more expensive.
## Scope
Bump `debezium` in `gradle/libs.versions.toml` from `3.4.3.Final` to `3.6.0.Final` and fix any resulting breakage.
**The connector set is unchanged.** Comparing the `debezium-bom` contents for 3.4.2 and 3.6.0, the only addition is `debezium-connector-common` (an internal refactor). No wrapper module gains or loses a target.
## Risks
The real risk is our coupling to Debezium **internals**, which are not API-stable across minor versions:
- `debezium/core` implements the schema-history SPI (`PulsarSchemaHistory`) against `io.debezium.storage` / `SchemaHistory` interfaces.
- `kafka-connect-adaptor` (`AbstractKafkaConnectSource`) starts Debezium tasks directly and depends on connector/task config-key conventions.
- `DebeziumSource` subclasses assert on Debezium's `connector.class` / `task.class` values.
Issue #61 is a concrete example of this coupling already breaking against 3.4.x (multi-partition SQL Server connectors never receive `task.id` because we bypass `taskConfigs()`). Expect similar sharp edges on a minor bump; #61 should ideally be fixed first, or at least understood, since it touches the same task-startup path.
## Prerequisites
The integration tests added in #58 (MongoDB), #60 (SQL Server), and #62 (Oracle) should be **merged before this migration**. Prior to those, a Debezium bump was validated only by the MySQL and Postgres tests. With all five source connectors exercised against real databases via Testcontainers, this upgrade either passes CI across the board or points directly at the connector that broke.
## Note: this will not fix the Testcontainers conflict
`debezium-bom:3.4.2.Final` pins Testcontainers core to `2.0.2`; `3.6.0.Final` (via `debezium-build-parent`) pins `2.0.3`. Both are 2.x, while the specialized Testcontainers modules (e.g. `org.testcontainers:oracle-free`) still target 1.21.x and reference shaded classes removed in 2.x. Debezium-module integration tests must keep using the `GenericContainer` pattern (as `debezium/mysql` and `debezium/oracle` do) regardless of this upgrade — see the discussion in #62.
## Suggested order
1. Merge #58, #60, #62 (integration-test safety net)
2. Land the `3.4.3.Final` patch bump (low risk, free fixes)
3. Address or characterize #61
4. Migrate to `3.6.0.Final` and let the five connector integration tests validate it
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in gradle/libs.versions.toml and inspect the Debezium version and BOM changes. Read PulsarSchemaHistory, AbstractKafkaConnectSource, and DebeziumSource, then run the MongoDB, SQL Server, Oracle, MySQL, and Postgres integration tests from #58, #60, and #62. Done means the migration reaches 3.6.0.Final, the connector set remains unchanged, and all five integrations pass without regressions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, build-system, testing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100