apache / apache/hudi

[SUPPORT] Hudi Clean and Delta commits taking ~50 mins to finish frequently

Open
#7,595 10 comments 0 reactions 1 assignee Claimed by @alexeykudinkin View on GitHub
area:metadata-table area:table-service priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

We have a Hudi table with metadata enabled and using delta streamer, async clean, async compact services . Delta commit and clean operations are taking ~50 minutes frequently.

A clear and concise description of the problem.

**To Reproduce**

Steps to reproduce the behavior:

1. Run 0.12.1 version Hudi with metadata table enabled.
2.Enable async compaction and cleaner services.
3. use the below configuration.
hoodie.cleaner.policy=KEEP_LATEST_COMMITS
hoodie.clean.automatic=true
hoodie.clean.async=true
hoodie.cleaner.commits.retained=5
hoodie.keep.min.commits=10
#compaction config
hoodie.datasource.compaction.async.enable=true
hoodie.parquet.small.file.limit=1048576
hoodie.compaction.target.io=50
hoodie.metadata.metrics.enable=true

hoodie.metadata.index.bloom.filter.enable=false
hoodie.metadata.index.column.stats.enable=false
hoodie.write.concurrency.mode=optimistic_concurrency_control
hoodie.cleaner.policy.failed.writes=LAZY
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.InProcessLockProvider
hoodie.write.lock.wait_time_ms=300000
4.

**Expected behavior**

Delta commit and clean actions should not take longer times.

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 0.12.1

* Spark version : 3.2.1

* Hive version : 2.3.5

* Hadoop version : 2.7.7

* Storage (HDFS/S3/GCS..) : GCS

* Running on Docker? (yes/no) : yes

**Additional context**

Add any other context about the problem here.

Delta streamer runs in continuous mode.

Adding the screenshot of timeline of operations

Screenshot 2023-01-03 at 11 50 00 AM

Screenshot 2023-01-03 at 11 50 57 AM

Below is the only error I see in logs.

**Stacktrace**

``` RequestHandler: Bad request response due to client view behind server view. Last known instant from client was 20230103113021745 but server has the following timeline [[20221128033016359__rollback__COMPLETED], [20221128042615784__rollback__COMPLETED], [20221128052249948__rollback__COMPLETED], [20221128100542977__rollback__COMPLETED], [20221128114411534__rollback__COMPLETED], [20221128121237952__rollback__COMPLETED], [20221128121547373__rollback__COMPLETED], [20221128124007294__rollback__COMPLETED], [20221128130510784__rollback__COMPLETED], [20221128150135765__rollback__COMPLETED], [20221202082857955__rollback__COMPLETED], [20221202083358380__rollback__COMPLETED], [20221205180609234__rollback__COMPLETED], [20221213024840399__rollback__COMPLETED], [20221215121336002__rollback__COMPLETED], [20230103075416732__clean__COMPLETED], [20230103080003681__clean__COMPLETED], [20230103080537813__clean__COMPLETED], [20230103081110194__clean__COMPLETED], [20230103081642791__clean__COMPLETED], [20230103082158513__clean__COMPLETED], [20230103082749103__clean__COMPLETED], [20230103083327661__clean__COMPLETED], [20230103083915577__clean__COMPLETED], [20230103084450294__clean__COMPLETED], [20230103085022170__clean__COMPLETED], [20230103085539296__deltacommit__COMPLETED], [20230103085550414__clean__COMPLETED], [20230103090129353__deltacommit__COMPLETED], [20230103090140117__clean__COMPLETED], [20230103090705599__deltacommit__COMPLETED], [20230103090716308__clean__COMPLETED], [20230103091245975__deltacommit__COMPLETED], [20230103091256846__clean__COMPLETED], [20230103091825253__deltacommit__COMPLETED], [20230103091836101__clean__COMPLETED], [20230103092403683__deltacommit__COMPLETED], [20230103092414824__clean__COMPLETED], [20230103092828723__commit__COMPLETED], [20230103092851264__clean__COMPLETED], [20230103092923310__deltacommit__COMPLETED], [20230103093158260__clean__COMPLETED], [20230103102048896__deltacommit__COMPLETED], [20230103102100480__clean__COMPLETED], [20230103102637434__deltacommit__COMPLETED], [20230103102648856__clean__COMPLETED], [20230103103218354__deltacommit__COMPLETED], [20230103103229738__clean__COMPLETED], [20230103103812033__deltacommit__COMPLETED], [20230103103823381__clean__COMPLETED], [20230103104351306__deltacommit__COMPLETED], [20230103104402684__clean__COMPLETED], [20230103104950491__deltacommit__COMPLETED], [20230103105002062__clean__COMPLETED], [20230103105541444__deltacommit__COMPLETED], [20230103105552964__clean__COMPLETED], [20230103110154035__deltacommit__COMPLETED], [20230103110205541__clean__COMPLETED], [20230103110749857__deltacommit__COMPLETED], [20230103110801657__clean__COMPLETED], [20230103111344582__deltacommit__COMPLETED], [20230103111356019__clean__COMPLETED], [20230103111912226__deltacommit__COMPLETED], [20230103111923380__clean__COMPLETED], [20230103112519397__deltacommit__COMPLETED], [20230103112531041__clean__COMPLETED], [20230103113021745__commit__COMPLETED], [20230103113045783__clean__COMPLETED]]```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.