apache / apache/iggy

feat(connectors): add Redis sink connector for Iggy streams

Open
#3,530 0 comments 0 reactions 1 assignee Claimed by @ryerraguntla View on GitHub
connectors enhancement rust
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Description

## Summary
Add a new Rust sink plugin (`redis_sink`) to the Iggy connectors runtime that consumes messages from Iggy streams/topics and writes them to Redis.
## Motivation
Redis is widely used as a low-latency cache, materialized view store, and event buffer (via Redis Streams). A native Redis sink enables pipelines such as:
- Iggy → Redis cache/materialized state
- Iggy → Redis Streams for downstream fan-out
- Event archival with optional TTL for cache use cases
There is currently no Redis sink in `core/connectors/sinks/`.
## Scope (v1)
Implement an initial sink with **Redis Streams** mode:
- `XADD` writes with pipelined batching
- Deterministic entry IDs for idempotent replay (pattern: `mongodb_sink` / `surrealdb_sink`)
- Configurable `connection_url` (`SecretString`), `stream_key`, `batch_size`, `payload_format`
- Optional Iggy metadata fields (`stream`, `topic`, `partition`, `offset`, `message_id`, headers)
- Transient vs permanent error mapping (connection errors retry; `WRONGTYPE`, auth failures do not)
- `open()` connectivity check (`PING`), `close()` stats logging

### Affected area / component

_No response_

### Proposed solution

### Suggested config (v1)
```toml
type = "sink"
key = "redis"
path = "target/release/libiggy_connector_redis_sink"
[plugin_config]
connection_url = "redis://localhost:6379"
mode = "stream"
stream_key = "iggy:events"
batch_size = 100
payload_format = "json"
include_metadata = true
max_retries = 3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.