apache / apache/hudi

Implement CDC data synchronization hudi based on Debezium Server

Open
#15,495 0 comments 0 reactions 0 assignees View on GitHub
area:writer from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

As of now, for debezium, we have a kafka based source. But we can try to leverage the debezium server directly and avoid additional maintanence costs. 

 

Ref:

 
# Does not rely on kafka connector, reducing maintenance costs,Flink cdc is more successful because it does not depend on kafka and is easy to use
# Support database or table level data synchronously written to hudi, support schema evolution
# Support flink or spark at the same time

This feature helps to expand the usage scenarios of hudi

[https://github.com/memiiso/debezium-server-spark]
[https://debezium.io/documentation/reference/operations/debezium-server.html#_implementation_of_a_new_sink]

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5073
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Debezium Server documentation for implementing a new sink and the linked memiiso/debezium-server-spark reference; no Hudi files or tests are named. Scope the design around database- or table-level synchronization into Hudi, schema evolution, and support for Flink or Spark, with completion confirmed against those stated goals.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.