apache / apache/iggy

feat(connectors): Implement QuestDB sink connector over QWP

Open
#4,086 1 comment 2 reactions 1 assignee Claimed by @javier View on GitHub
connectors
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Description

As part of the connector ecosystem roadmap (#2753) I would like to add a native Rust **QuestDB sink** under `core/connectors/sinks/questdb_sink`.

The roadmap lists QuestDB at P3, noting that the InfluxDB connector could cover it via line protocol. That is now out of date: QuestDB 10 introduced **QWP (the QuestDB Wire Protocol)**, a binary columnar protocol over WebSocket which is the native ingestion path for every QuestDB client. ILP remains only for InfluxDB compatibility, so routing through it gives up typed columnar encoding, per-flush acknowledgements, client-side store-and-forward, multi-host failover and durable acknowledgements.

I work on QuestDB, so I can test the Enterprise-only paths directly: TLS, buffering across a failover, and durable ACK.

### Affected area / component

Connectors

### Proposed solution

A `Sink` plugin built on the official [`questdb-rs`](https://crates.io/crates/questdb-rs) crate (Apache-2.0), speaking QWP over WebSocket. One crate serves QuestDB open source and Enterprise, since credentials, TLS, store-and-forward, reconnect and failover are all selected through connect-string keys.

Configuration is a QuestDB connect string plus a small mapping layer (target table, designated-timestamp source, and which payload fields become `SYMBOL` or `UUID` columns), following the `influxdb_sink` / `postgres_sink` conventions.

Delivery is at-least-once, so tables need `DEDUP UPSERT KEYS(...)`. Worth flagging for reviewers: because the runtime commits offsets before `consume()` runs and discards its result (#2928, #2927), a failed batch cannot be replayed from Apache Iggy. QWP's on-disk store-and-forward covers that gap on the write path, which is a concrete reason to prefer it over ILP-over-HTTP here.

Scope is the sink only.

I have a small proof of concept working end to end against QuestDB 10 and QuestDB Enterprise, with unit and `testcontainers` integration coverage. Happy to adjust the design before turning it into a PR.

### Alternatives considered

_No response_

### Contribution

- [x] I'm willing to submit a pull request to implement this feature

### Good first issue

- [ ] I think this could be a good first issue for a new contributor

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.