Add Apache Hudi Sink Connector
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
### Description
Add an Apache Hudi Sink Connector to Iggy Connect.
The connector should consume messages from Apache Iggy and continuously write them to Apache Hudi tables, enabling Iggy streams to be used as a real-time ingestion layer for lakehouse and analytical workloads.
Motivation
----
Apache Hudi is widely used for building transactional data lakes and supports incremental processing, upserts, deletes, CDC, and large-scale analytics.
A native Hudi Sink Connector would enable use cases such as:
* Iggy streams → Hudi / data lake
* CDC pipelines
* Real-time analytics ingestion
* Application events and logs archival
* AI/ML data pipelines
* Operational data → lakehouse
* Long-term storage of Iggy streams
```
Applications
|
v
Apache Iggy
|
v
Hudi Sink Connector
|
v
Apache Hudi
|
v
S3 / GCS / Azure / Data Lake
```
Requirements
---
The connector should follow the existing Iggy Connect sink connector architecture and conventions.
At minimum:
* Consume messages from configured Iggy streams/topics
* Write records to Apache Hudi tables
* Support configurable record keys
* Support configurable partition fields
* Support batched writes
* Support inserts and upserts
* Correctly commit Iggy offsets after successful writes
* Handle retries and write failures
* Support graceful restart/recovery
* Add configuration validation
* Add unit tests
* Add integration/end-to-end tests
* Add a runnable example
* Update Iggy Connect documentation
* Add Apache Hudi to the supported connectors documentation
Testing
---
Add an end-to-end test covering:
`Producer → Apache Iggy → Hudi Sink Connector → Hudi Table`
Verify message ingestion, batching, upserts, offset handling, and recovery after connector restart.
### Affected area / component
_No response_
### Proposed solution
_No response_
### Alternatives considered
_No response_
### Contribution
- [ ] I'm willing to submit a pull request to implement this feature
### Good first issue
- [ ] I think this could be a good first issue for a new contributor
Contributor guide
Assessment
This issue has not been assessed yet.