stackabletech / stackabletech/kafka-operator

Support for replication using MirrorMaker 2

Open
#693 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

customer-request
Dominant language
Rust
Stars
28
Forks
8
Avg merge
2d 12h
Merged PRs (30d)
9

Description

Run MirrorMaker 2 as a Stacklet

We do have NiFi in the stack which can be used for most data moving operations but it'd be overhead if all you need is to mirror two Kafkas.
So, this is about adding native support for MirrorMaker2 to SDP.

This is also useful for migrating an existing Kafka into Stackable.

FYI: We have internal notes from #923 in docs/modules/kafka/examples/mirror_maker/ which might or might not help but need updating with this either way.

Scope

  • A CRD that runs MirrorMaker 2 in dedicated mode, replicating between Kafka clusters
    • See the open questions below
  • Docs and an integration test replicating between two SDP Kafka clusters.
  • Update/delete docs/modules/kafka/examples/mirror_maker/

Expected effort

10 days.
If it takes longer, stop and let me know.

Out of scope

  • Kafka Connect
  • Per-connector actions: restart, pause, resume, stop, offset rewind

Open questions

  • CRD name (MirrorMakerCluster?)
  • Do we need Discovery 2.0 for this?
  • How elaborate is the CRD: do we model the connector options as real structs or is all of that done using overrides.
  • dedicated.mode.enable.internal.rest defaults to false and is required above one replica. Always on, or only when replicas exceed one?
  • Kerberos?
  • Both directions in one CR, or one CR per direction?

Possible approach

Strimzi implements this differently than I would and Claude believes that this is because of KAFKA-9981 which was still open at the time they implemented it and which would make the dedicated mode I propose here unusable. They are building a Kafka Connect cluster with MirrorMaker config.
I propose we use the dedicated MirrorMaker mode.

Kafka Connect is purely configured using a REST API so we'd need to have the whole access delegation story and a client in the operator to talk to connect to configure it etc.
MirrorMaker can be configured using a file. That seems way more convenient.

Downside(ish): Connect exposes a web server with read & write stuff (status and pause/resume etc.). MirrorMaker does not. But we could easily patch those lines in. The patch looks simple. I don't want that for v1 though.

This is the direction I'd prefer. If you disagree tell me before implementation starts.

Notes

[!NOTE]
These are notes that came out of the Claude research

  • Connect and MirrorMaker 2 are already in our images
  • In dedicated mode the properties file is the source of truth. MirrorHerder extends
    DistributedHerder and, on becoming leader, diffs the desired connector configs against the
    config topic and replaces them. A changed properties file therefore takes effect on restart or leader change which I believe matches our commons operator restarter pattern
  • One JVM holds a Worker and a herder per replication flow, so several flows do not require
    several sets of pods.
  • Connector and task state is available as a metric (connector-metrics and
    connector-task-metrics, attribute status), which covers most of what a CR status would want
    without any REST API. The failure trace is not a metric; it is in status.storage.topic.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing notes in docs/modules/kafka/examples/mirror_maker/ and the issue's open questions about the CRD, dedicated mode, discovery, authentication, and replication direction. Then inspect how the operator defines CRDs, deploys workloads, and writes integration tests. Done means native dedicated-mode MirrorMaker 2 support, updated or removed example documentation, and an integration test replicating between two SDP Kafka clusters.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust
Domain
devops, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.