[Bug] Sink multi-parallelism error hive lock
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
1.1.1
### Compute Engine
flink 1.20.1
### Minimal reproduce step
CREATE TABLE t_changelog_input (
age BIGINT,
money BIGINT,
name STRING,
PRIMARY KEY (name) NOT ENFORCED
)WITH (
"bucket" = "4",
'file.compression' = 'snappy',
'merge-engine' = 'deduplicate',
'changelog-producer' = 'input',
"sink.parallelism" = "4"
);
The table is created as above. When a large amount of data is written or a large amount of data is changed, the following error will appear. However, when sink.parallelism is 1, it will not appear. My data is stored on S3 and uses hivemetastore 3.1.2.Is there something wrong with my usage? Is there any way to increase the write parallelism?
### What doesn't meet your expectations?
I hope that even when the data is stored on S3, which does not have atomic storage, it can still support multiple concurrent writes.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the CREATE TABLE and write workload with sink.parallelism set to 1 and 4 on S3, using Flink 1.20.1 and Hive Metastore 3.1.2, and capture the full Hive-lock error shown in the attachment. Trace the concurrent-write and locking path to determine whether parallel writes are supported; done means the failure has a confirmed cause and a verified fix or documented limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100