feat(connectors): Iggy Source/Sink Connector: Native Cluster-to-Cluster Replication
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
### Description
**Summary**
Implement an Iggy Source/Sink Connector that consumes data from an upstream Iggy cluster and publishes it into a downstream Iggy cluster.
This enables seamless cluster-to-cluster replication and provides the foundation for multi-region deployments, disaster recovery, active-active architectures, edge-to-cloud streaming, and hierarchical streaming topologies.
Unlike traditional application-level replication, this connector allows operators to replicate selected streams, partitions, or namespaces without requiring producers or consumers to change.
**Motivation**
Many production deployments require data to exist in multiple locations.
Today, applications would need to publish to multiple Iggy clusters directly, increasing complexity and coupling replication logic into business applications.
A native Iggy Source/Sink Connector would make replication a first-class capability by continuously consuming from one cluster and publishing to another while preserving ordering and delivery guarantees.
Producers
│
▼
+------------------+
| Iggy Cluster A |
| (Primary) |
+------------------+
│
Iggy Source/Sink Connector
│
▼
+------------------+
| Iggy Cluster B |
| (Replica) |
+------------------+
│
Local Consumers
The connector should behave like any other Iggy connector:
* Source = upstream Iggy
* Sink = downstream Iggy
Aspects to cover (ideal, not a comprehensive list, can be broken down into follow-up tasks after the basic version is in place):
* Iggy → Iggy replication
* configurable source streams
* configurable destination streams
* partition-aware replication
* offset checkpointing
* resumable replication
* configurable batching
* configurable compression
* TLS support
* authentication support
* metrics
* retry with backoff
* configurable parallelism
### Affected area / component
Connectors
### Proposed solution
_No response_
### Alternatives considered
_No response_
### Contribution
- [ ] I'm willing to submit a pull request to implement this feature
### Good first issue
- [x] I think this could be a good first issue for a new contributor
Contributor guide
Assessment
This issue has not been assessed yet.