apache / apache/pulsar-connectors
[feat][io] Add Debezium Spanner source connector
- Dominant language
- Java
- Stars
- 26
- Forks
- 25
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 1
Description
## Motivation
Debezium ships a Google Cloud Spanner source connector (`io.debezium:debezium-connector-spanner`, present in the `debezium-bom` at our pinned version), but this repository does not wrap it. We currently wrap 5 of Debezium's 12 source connectors: MySQL, PostgreSQL, MongoDB, Oracle, and SQL Server.
Spanner is a managed, horizontally-scaled database with a growing Pulsar user base, and the connector itself wraps cleanly — the interesting work is in testing.
## Proposal
Add a `debezium/spanner` module (`pulsar-io-debezium-spanner`) following the structure of the existing Debezium modules:
- `DebeziumSpannerSource` extending `DebeziumSource`
- `include("debezium:pulsar-io-debezium-spanner")` in `settings.gradle.kts`
- `debezium-connector-spanner = { module = "io.debezium:debezium-connector-spanner" }` in the version catalog
- NAR packaging + distribution wiring
- An integration test against the **Spanner emulator** (`gcr.io/cloud-spanner-emulator/emulator`) so CI needs no GCP credentials
## Notes
- Spanner CDC works via **change streams**, which must be created with DDL (`CREATE CHANGE STREAM ...`) as part of test setup. Confirm the emulator supports change streams at the version we target — if it does not, the test must be credential-gated, and we should avoid repeating the `azure-data-explorer` situation (#49) where a credential-gated E2E test means the connector is never actually exercised in CI.
- Spanner is a **multi-partition** connector, so it will likely hit the `task.id` gap described in #61. That issue should land first.
- Use the `GenericContainer` pattern (see the Testcontainers version conflict noted in #62).
_Identified by a scan of the Debezium connector catalog against our connector inventory._
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the existing Debezium connector modules, then inspect settings.gradle.kts, the version catalog, and the DebeziumSpannerSource entry point. Verify whether gcr.io/cloud-spanner-emulator/emulator supports change streams at the target version and review #61 and #62 before implementing. Done means the module is packaged and distributed, wired into the build, and exercised by an emulator-backed integration test in CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, java
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100