apache / apache/paimon

[Bug] HiveCatalogLock runWithLock() method executed exception :“Acquire lock failed with time: PT8M7.098S”

Open
#6,540 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

Paimon-1.1.1

### Compute Engine

Flink-1.18.1

### Minimal reproduce step

No recurrence has been reported.

### What doesn't meet your expectations?

```
Java.lang.RuntimeException: Exception occurs when committing snapshot #29953 by user dca7f3a6-3177-4ae2-b9de-0902024b5314 with identifier 1 and kind APPEND.
Cannot clean up because we can't determine the success.
at org.apache.paimon.operation.FileStoreCommitImpl.commitSnapshotImpl(FileStoreCommitImpl.java:1166)
at org.apache.paimon.operation.FileStoreCommitImpl.tryCommitOnce(FileStoreCommitImpl.java:1015)
at org.apache.paimon.operation.FileStoreCommitImpl.tryCommit(FileStoreCommitImpl.java:732)
at org.apache.paimon.operation.FileStoreCommitImpl.commit(FileStoreCommitImpl.java:323)
at org.apache.paimon.table.sink.TableCommitImpl.commitMultiple(TableCommitImpl.java:218)
at org.apache.paimon.table.sink.TableCommitImpl.filterAndCommitMultiple(TableCommitImpl.java:257)
at org.apache.paimon.flink.sink.StoreCommitter.filterAndCommit(StoreCommitter.java:119)
at org.apache.paimon.flink.sink.Committer.filterAndCommit(Committer.java:60)
at org.apache.paimon.flink.sink.RestoreAndFailCommittableStateManager.recover(RestoreAndFailCommittableStateManager.java:82)
at org.apache.paimon.flink.sink.RestoreAndFailCommittableStateManager.initializeState(RestoreAndFailCommittableStateManager.java:77)
at org.apache.paimon.flink.sink.CommitterOperator.initializeState(CommitterOperator.java:147)
at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:274)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:753)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:728)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:693)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:922)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: Acquire lock failed with time: PT8M1.606S
at org.apache.paimon.hive.HiveCatalogLock.lock(HiveCatalogLock.java:110)
at org.apache.paimon.hive.HiveCatalogLock.runWithLock(HiveCatalogLock.java:66)
at org.apache.paimon.operation.Lock$CatalogLockImpl.runWithLock(Lock.java:73)
at org.apache.paimon.catalog.RenamingSnapshotCommit.commit(RenamingSnapshotCommit.java:69)
at org.apache.paimon.operation.FileStoreCommitImpl.commitSnapshotImpl(FileStoreCommitImpl.java:1161)
... 22 more
```

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with HiveCatalogLock.runWithLock() and lock() at HiveCatalogLock.java:66 and :110, then trace the failure through RenamingSnapshotCommit.commit and FileStoreCommitImpl.commitSnapshotImpl. Reproduce or otherwise isolate the lock-acquisition failure using the Paimon 1.1.1 and Flink 1.18.1 context; done means the cause and an appropriate validated fix are established.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.