apache / apache/iggy

Add Apache Hudi Sink Connector

Open
#4,002 3 comments 0 reactions 1 assignee Claimed by @mattp5657 View on GitHub
connectors
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Description

Add an Apache Hudi Sink Connector to Iggy Connect.

The connector should consume messages from Apache Iggy and continuously write them to Apache Hudi tables, enabling Iggy streams to be used as a real-time ingestion layer for lakehouse and analytical workloads.

Motivation
----
Apache Hudi is widely used for building transactional data lakes and supports incremental processing, upserts, deletes, CDC, and large-scale analytics.

A native Hudi Sink Connector would enable use cases such as:

* Iggy streams → Hudi / data lake
* CDC pipelines
* Real-time analytics ingestion
* Application events and logs archival
* AI/ML data pipelines
* Operational data → lakehouse
* Long-term storage of Iggy streams
```
Applications
|
v
Apache Iggy
|
v
Hudi Sink Connector
|
v
Apache Hudi
|
v
S3 / GCS / Azure / Data Lake
```

Requirements
---
The connector should follow the existing Iggy Connect sink connector architecture and conventions.

At minimum:

* Consume messages from configured Iggy streams/topics
* Write records to Apache Hudi tables
* Support configurable record keys
* Support configurable partition fields
* Support batched writes
* Support inserts and upserts
* Correctly commit Iggy offsets after successful writes
* Handle retries and write failures
* Support graceful restart/recovery
* Add configuration validation
* Add unit tests
* Add integration/end-to-end tests
* Add a runnable example
* Update Iggy Connect documentation
* Add Apache Hudi to the supported connectors documentation

Testing
---
Add an end-to-end test covering:
`Producer → Apache Iggy → Hudi Sink Connector → Hudi Table`
Verify message ingestion, batching, upserts, offset handling, and recovery after connector restart.

### Affected area / component

_No response_

### Proposed solution

_No response_

### Alternatives considered

_No response_

### Contribution

- [ ] I'm willing to submit a pull request to implement this feature

### Good first issue

- [ ] I think this could be a good first issue for a new contributor

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.