[SUPPORT] How to do Schema Evolution with Apache Flink DataStream API when doing CDC?
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I have an Apache Flink application, that is consuming directly from a database using Flink CDC Connectors, however, I am not able to find any documentation on how to manage when a table schema evolves, when writing to Hudi. The only Apache Flink Sink that is showed in the documentation is the Hoodie Pipeline builder and its for rowData. Is there an Avro Sink?
How could one build this, so the Flink Application doesnt have to be stopped and is able to dnamically evolve schema in the hudi tables??
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Flink CDC Connectors and Hoodie Pipeline builder documentation, focusing on the RowData path and whether an Avro sink is available. Done means establishing a documented or working approach for evolving database schemas while writing to Hudi without stopping the Flink application.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100