apache / apache/hudi

[SUPPORT] deltastreamer support migrate COW table to MOR

Open
#8,249 2 comments 0 reactions 0 assignees View on GitHub
area:ingest priority:medium type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? yes

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org. sent email please help involve me in

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

My company is using deltastreamer to ingestion data from kafka to hdfs, the old hudi table type is COW. But the write latency of the COW table is much higher than MOR table. So we are going to migrate the COW tables to MOR.
According to the [FAQ](https://hudi.apache.org/docs/faq/#how-to-convert-an-existing-cow-table-to-mor), we can change the existing COW table to MOR by just changing the `hoodity.table.type` property.
But there is one issue for continuing deltastreamer of MOR table on the existing path, the checkpoint from old COW table will lost, so there may be dataloss in such cases.

I find the cause of the checkpoint loss.
In the refreshTimeline method, when table is MOR only get checkpoint from deltacommits, that's why the checkpoint loss when migrating COW to MOR https://github.com/apache/hudi/blob/ce21873f332f1728ad46aeb066777e49456ef522/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java#L325

Simply put, I want the deltastreamer to support migrate hudi table directly by a parameter

**Expected behavior**

deltastreamer offer a parameter `--migration-type` to support migrate existing COW table to MOR,
the value of the param `--migration-type` can be:
1. NONE (default no migration)
2. COW_TO_MOR
3. MOR_TO_COW (TODO)

I had raised a PR #8247, maybe someone can help review, thanks

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java and the refreshTimeline method identified in the issue. Review PR #8247 and the existing checkpoint handling before assessing the proposed --migration-type values. Done means DeltaStreamer can continue from an existing COW table when migrating to MOR without losing its checkpoint.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.