[SUPPORT] Hudi Clean and Delta commits taking ~50 mins to finish frequently
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
We have a Hudi table with metadata enabled and using delta streamer, async clean, async compact services . Delta commit and clean operations are taking ~50 minutes frequently.
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1. Run 0.12.1 version Hudi with metadata table enabled.
2.Enable async compaction and cleaner services.
3. use the below configuration.
hoodie.cleaner.policy=KEEP_LATEST_COMMITS
hoodie.clean.automatic=true
hoodie.clean.async=true
hoodie.cleaner.commits.retained=5
hoodie.keep.min.commits=10
#compaction config
hoodie.datasource.compaction.async.enable=true
hoodie.parquet.small.file.limit=1048576
hoodie.compaction.target.io=50
hoodie.metadata.metrics.enable=true
hoodie.metadata.index.bloom.filter.enable=false
hoodie.metadata.index.column.stats.enable=false
hoodie.write.concurrency.mode=optimistic_concurrency_control
hoodie.cleaner.policy.failed.writes=LAZY
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider
hoodie.write.lock.wait_time_ms=300000
4.
**Expected behavior**
Delta commit and clean actions should not take longer times.
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.12.1
* Spark version : 3.2.1
* Hive version : 2.3.5
* Hadoop version : 2.7.7
* Storage (HDFS/S3/GCS..) : GCS
* Running on Docker? (yes/no) : yes
**Additional context**
Add any other context about the problem here.
Delta streamer runs in continuous mode.
Adding the screenshot of timeline of operations


Below is the only error I see in logs.
**Stacktrace**
``` RequestHandler: Bad request response due to client view behind server view. Last known instant from client was 20230103113021745 but server has the following timeline [[20221128033016359__rollback__COMPLETED], [20221128042615784__rollback__COMPLETED], [20221128052249948__rollback__COMPLETED], [20221128100542977__rollback__COMPLETED], [20221128114411534__rollback__COMPLETED], [20221128121237952__rollback__COMPLETED], [20221128121547373__rollback__COMPLETED], [20221128124007294__rollback__COMPLETED], [20221128130510784__rollback__COMPLETED], [20221128150135765__rollback__COMPLETED], [20221202082857955__rollback__COMPLETED], [20221202083358380__rollback__COMPLETED], [20221205180609234__rollback__COMPLETED], [20221213024840399__rollback__COMPLETED], [20221215121336002__rollback__COMPLETED], [20230103075416732__clean__COMPLETED], [20230103080003681__clean__COMPLETED], [20230103080537813__clean__COMPLETED], [20230103081110194__clean__COMPLETED], [20230103081642791__clean__COMPLETED], [20230103082158513__clean__COMPLETED], [20230103082749103__clean__COMPLETED], [20230103083327661__clean__COMPLETED], [20230103083915577__clean__COMPLETED], [20230103084450294__clean__COMPLETED], [20230103085022170__clean__COMPLETED], [20230103085539296__deltacommit__COMPLETED], [20230103085550414__clean__COMPLETED], [20230103090129353__deltacommit__COMPLETED], [20230103090140117__clean__COMPLETED], [20230103090705599__deltacommit__COMPLETED], [20230103090716308__clean__COMPLETED], [20230103091245975__deltacommit__COMPLETED], [20230103091256846__clean__COMPLETED], [20230103091825253__deltacommit__COMPLETED], [20230103091836101__clean__COMPLETED], [20230103092403683__deltacommit__COMPLETED], [20230103092414824__clean__COMPLETED], [20230103092828723__commit__COMPLETED], [20230103092851264__clean__COMPLETED], [20230103092923310__deltacommit__COMPLETED], [20230103093158260__clean__COMPLETED], [20230103102048896__deltacommit__COMPLETED], [20230103102100480__clean__COMPLETED], [20230103102637434__deltacommit__COMPLETED], [20230103102648856__clean__COMPLETED], [20230103103218354__deltacommit__COMPLETED], [20230103103229738__clean__COMPLETED], [20230103103812033__deltacommit__COMPLETED], [20230103103823381__clean__COMPLETED], [20230103104351306__deltacommit__COMPLETED], [20230103104402684__clean__COMPLETED], [20230103104950491__deltacommit__COMPLETED], [20230103105002062__clean__COMPLETED], [20230103105541444__deltacommit__COMPLETED], [20230103105552964__clean__COMPLETED], [20230103110154035__deltacommit__COMPLETED], [20230103110205541__clean__COMPLETED], [20230103110749857__deltacommit__COMPLETED], [20230103110801657__clean__COMPLETED], [20230103111344582__deltacommit__COMPLETED], [20230103111356019__clean__COMPLETED], [20230103111912226__deltacommit__COMPLETED], [20230103111923380__clean__COMPLETED], [20230103112519397__deltacommit__COMPLETED], [20230103112531041__clean__COMPLETED], [20230103113021745__commit__COMPLETED], [20230103113045783__clean__COMPLETED]]```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.