stackabletech / stackabletech/kafka-operator
Support for replication using MirrorMaker 2
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 28
- Forks
- 8
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 9
Description
Run MirrorMaker 2 as a Stacklet
We do have NiFi in the stack which can be used for most data moving operations but it'd be overhead if all you need is to mirror two Kafkas.
So, this is about adding native support for MirrorMaker2 to SDP.
This is also useful for migrating an existing Kafka into Stackable.
FYI: We have internal notes from #923 in docs/modules/kafka/examples/mirror_maker/ which might or might not help but need updating with this either way.
Scope
- A CRD that runs MirrorMaker 2 in dedicated mode, replicating between Kafka clusters
- See the open questions below
- Docs and an integration test replicating between two SDP Kafka clusters.
- Update/delete
docs/modules/kafka/examples/mirror_maker/
Expected effort
10 days.
If it takes longer, stop and let me know.
Out of scope
- Kafka Connect
- Per-connector actions: restart, pause, resume, stop, offset rewind
Open questions
- CRD name (MirrorMakerCluster?)
- Do we need Discovery 2.0 for this?
- How elaborate is the CRD: do we model the connector options as real structs or is all of that done using overrides.
dedicated.mode.enable.internal.restdefaults to false and is required above one replica. Always on, or only when replicas exceed one?- Kerberos?
- Both directions in one CR, or one CR per direction?
Possible approach
Strimzi implements this differently than I would and Claude believes that this is because of KAFKA-9981 which was still open at the time they implemented it and which would make the dedicated mode I propose here unusable. They are building a Kafka Connect cluster with MirrorMaker config.
I propose we use the dedicated MirrorMaker mode.
Kafka Connect is purely configured using a REST API so we'd need to have the whole access delegation story and a client in the operator to talk to connect to configure it etc.
MirrorMaker can be configured using a file. That seems way more convenient.
Downside(ish): Connect exposes a web server with read & write stuff (status and pause/resume etc.). MirrorMaker does not. But we could easily patch those lines in. The patch looks simple. I don't want that for v1 though.
This is the direction I'd prefer. If you disagree tell me before implementation starts.
Notes
[!NOTE]
These are notes that came out of the Claude research
- Connect and MirrorMaker 2 are already in our images
- In dedicated mode the properties file is the source of truth.
MirrorHerderextends
DistributedHerderand, on becoming leader, diffs the desired connector configs against the
config topic and replaces them. A changed properties file therefore takes effect on restart or leader change which I believe matches our commons operator restarter pattern - One JVM holds a
Workerand a herder per replication flow, so several flows do not require
several sets of pods. - Connector and task state is available as a metric (
connector-metricsand
connector-task-metrics, attributestatus), which covers most of what a CR status would want
without any REST API. The failure trace is not a metric; it is instatus.storage.topic.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing notes in docs/modules/kafka/examples/mirror_maker/ and the issue's open questions about the CRD, dedicated mode, discovery, authentication, and replication direction. Then inspect how the operator defines CRDs, deploys workloads, and writes integration tests. Done means native dedicated-mode MirrorMaker 2 support, updated or removed example documentation, and an integration test replicating between two SDP Kafka clusters.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust
- Domain
- devops, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100