Implement CDC data synchronization hudi based on Debezium Server
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
As of now, for debezium, we have a kafka based source. But we can try to leverage the debezium server directly and avoid additional maintanence costs.
Ref:
# Does not rely on kafka connector, reducing maintenance costs,Flink cdc is more successful because it does not depend on kafka and is easy to use
# Support database or table level data synchronously written to hudi, support schema evolution
# Support flink or spark at the same time
This feature helps to expand the usage scenarios of hudi
[https://github.com/memiiso/debezium-server-spark]
[https://debezium.io/documentation/reference/operations/debezium-server.html#_implementation_of_a_new_sink]
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5073
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Debezium Server documentation for implementing a new sink and the linked memiiso/debezium-server-spark reference; no Hudi files or tests are named. Scope the design around database- or table-level synchronization into Hudi, schema evolution, and support for Flink or Spark, with completion confirmed against those stated goals.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100