Support running the Spanner source connector without a Kafka broker
- Dominant language
- HTML
- Stars
- 6
- Forks
- 8
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 1
Description
## Feature request or enhancement
_For feature requests or enhancements, please provide the following information:_
**Which use case/requirement will be addressed by the proposed feature?**
While most Debezium source connectors can run in the embedded engine / Debezium Server without a Kafka broker, the Spanner source connector currently requires one. It uses Kafka internally for cross task coordination and low watermark aggregation which makes it hard to use in broker-less deployments.
We've been exploring making it runnable without a broker for the single task case, as with one task there's no cross task state to share, so neither the coordination nor the low watermark aggregation needs a broker.
The approach pulls the Kafka-coupled coordination behind a small SPI. This keeps the existing Kafka path as the default (multi-task, unchanged) and adds a broker-less single task implementation (as a nice side effect this turns coordination into a pluggable extension point, so others could provide their own backend instead of Kafka). The mode is chosen by a simple config flag.
We have a working PoC but before going further I wanted to ask: would the maintainers be open to upstreaming something like this?
Happy to share the design, the SPI, and more details around the same.
Contributor guide
Research direction
Start by reviewing the proposed SPI, broker-less single-task mode, and configuration flag described in the issue; no repository files or tests are named. Clarify how coordination and low-watermark aggregation should work without Kafka while preserving the existing multi-task path. Done means there is an agreed upstreamable design and appropriate implementation tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kafka
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100