apache / apache/iggy

feat(connectors): Iggy source connector implementation

Open
#3,869 0 comments 0 reactions 0 assignees View on GitHub
connectors enhancement rust
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

Parent issue: #3764

Implement a source connector module that consumes messages from an upstream Iggy (cluster) into the connectors runtime. This issue serves as the overall implementation tracker. Please refer to the original proposal for detailed design.

## Milestones

### M1: Basic functionality
- Messages flow from upstream Iggy (source) through the connector to downstream Iggy
- Configurable stream and topic selection
- Progress persistence with resumable offset tracking

### M2: Fine-grained configuration
- TLS support
- Additional authentication methods (username/password, token)
- Flow control: finer-grained concurrency control and batching

### M3: Enhancement
- Compression and pro-active retry
- Partition-aware offset tracking

Contributor guide

Open the contributing guide

Research direction

Start with parent issue #3764 and the original proposal, since this tracker names no files, tests, or entry points. Map the connectors runtime and the upstream Iggy source flow, then use the milestones as the completion checklist: message flow, selection, resumable offsets, followed by TLS, authentication, flow control, compression, retries, and partition-aware tracking.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.