apache / apache/iggy

feat(connectors): Iggy Source/Sink Connector: Native Cluster-to-Cluster Replication

Open
#3,764 8 comments 0 reactions 1 assignee Claimed by @jiengup View on GitHub
connectors good first issue
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Description

**Summary**

Implement an Iggy Source/Sink Connector that consumes data from an upstream Iggy cluster and publishes it into a downstream Iggy cluster.

This enables seamless cluster-to-cluster replication and provides the foundation for multi-region deployments, disaster recovery, active-active architectures, edge-to-cloud streaming, and hierarchical streaming topologies.

Unlike traditional application-level replication, this connector allows operators to replicate selected streams, partitions, or namespaces without requiring producers or consumers to change.

**Motivation**

Many production deployments require data to exist in multiple locations.

Today, applications would need to publish to multiple Iggy clusters directly, increasing complexity and coupling replication logic into business applications.

A native Iggy Source/Sink Connector would make replication a first-class capability by continuously consuming from one cluster and publishing to another while preserving ordering and delivery guarantees.

Producers


+------------------+
| Iggy Cluster A |
| (Primary) |
+------------------+

Iggy Source/Sink Connector


+------------------+
| Iggy Cluster B |
| (Replica) |
+------------------+

Local Consumers

The connector should behave like any other Iggy connector:

* Source = upstream Iggy
* Sink = downstream Iggy

Aspects to cover (ideal, not a comprehensive list, can be broken down into follow-up tasks after the basic version is in place):
* Iggy → Iggy replication
* configurable source streams
* configurable destination streams
* partition-aware replication
* offset checkpointing
* resumable replication
* configurable batching
* configurable compression
* TLS support
* authentication support
* metrics
* retry with backoff
* configurable parallelism

### Affected area / component

Connectors

### Proposed solution

_No response_

### Alternatives considered

_No response_

### Contribution

- [ ] I'm willing to submit a pull request to implement this feature

### Good first issue

- [x] I think this could be a good first issue for a new contributor

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.