[SUPPORT] deltastreamer support migrate COW table to MOR
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? yes
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org. sent email please help involve me in
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
My company is using deltastreamer to ingestion data from kafka to hdfs, the old hudi table type is COW. But the write latency of the COW table is much higher than MOR table. So we are going to migrate the COW tables to MOR.
According to the [FAQ](https://hudi.apache.org/docs/faq/#how-to-convert-an-existing-cow-table-to-mor), we can change the existing COW table to MOR by just changing the `hoodity.table.type` property.
But there is one issue for continuing deltastreamer of MOR table on the existing path, the checkpoint from old COW table will lost, so there may be dataloss in such cases.
I find the cause of the checkpoint loss.
In the refreshTimeline method, when table is MOR only get checkpoint from deltacommits, that's why the checkpoint loss when migrating COW to MOR https://github.com/apache/hudi/blob/ce21873f332f1728ad46aeb066777e49456ef522/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java#L325
Simply put, I want the deltastreamer to support migrate hudi table directly by a parameter
**Expected behavior**
deltastreamer offer a parameter `--migration-type` to support migrate existing COW table to MOR,
the value of the param `--migration-type` can be:
1. NONE (default no migration)
2. COW_TO_MOR
3. MOR_TO_COW (TODO)
I had raised a PR #8247, maybe someone can help review, thanks
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java and the refreshTimeline method identified in the issue. Review PR #8247 and the existing checkpoint handling before assessing the proposed --migration-type values. Done means DeltaStreamer can continue from an existing COW table when migrating to MOR without losing its checkpoint.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100