apache / apache/iggy

feat(connector): Add Opensearch source connector

Open
#3,503 5 comments 0 reactions 0 assignees View on GitHub
connectors
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Description

Add a Opensearch source connector that polls documents from a Opensearch index and produces JSON messages into Iggy.

Users should be able to ingest Opensearch documents into Iggy streams with stable cursor-based polling and persisted connector state.

### Affected area / component

Connectors

### Proposed solution

Scope
Add the source connector crate and sample runtime config.
Document source configuration, retry behavior, and primary-key cursor requirements.
Add Docker-backed integration coverage for producing indexed documents.

### Alternatives considered

_No response_

### Contribution

- [ ] I'm willing to submit a pull request to implement this feature

### Good first issue

- [ ] I think this could be a good first issue for a new contributor

Contributor guide

Open the contributing guide

Research direction

Begin in the Connectors area by comparing existing source connectors and their runtime configuration patterns. Add the source connector crate and sample configuration, document source settings, retries, and primary-key cursor requirements, then run the Docker-backed integration coverage to verify indexed documents are produced into Iggy with persisted state.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.