apache / apache/paimon

[Feature] Execute ManifestFileMeta#merge asynchronously to reduce the time of snapshot commit.

Open
#2,483 2 comments 0 reactions 1 assignee Claimed by @liming30 View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

In the snapshot commit phase, `ManifestFileMeta` will be merged. Consider this situation: when a partition expires, `fullCompaction` will be triggered according to the merge algorithm. At this time, all historical `ManifestEntry` will be rewritten into the new `ManifestFile`, and the time will increase with the number of `ManifestEntry`.

I tested about 20 * 40000 `ManifestEntry` and it took about 5 minutes.

### Solution

I think the merge of `ManifestFileMeta` can also be executed asynchronously like compaction.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.