[Feature] Execute ManifestFileMeta#merge asynchronously to reduce the time of snapshot commit.
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
In the snapshot commit phase, `ManifestFileMeta` will be merged. Consider this situation: when a partition expires, `fullCompaction` will be triggered according to the merge algorithm. At this time, all historical `ManifestEntry` will be rewritten into the new `ManifestFile`, and the time will increase with the number of `ManifestEntry`.
I tested about 20 * 40000 `ManifestEntry` and it took about 5 minutes.
### Solution
I think the merge of `ManifestFileMeta` can also be executed asynchronously like compaction.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.