apache / apache/paimon

[Bug] Sink multi-parallelism error hive lock

Open
#6,251 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

1.1.1

### Compute Engine

flink 1.20.1

### Minimal reproduce step

CREATE TABLE t_changelog_input (
age BIGINT,
money BIGINT,
name STRING,
PRIMARY KEY (name) NOT ENFORCED
)WITH (
"bucket" = "4",
'file.compression' = 'snappy',
'merge-engine' = 'deduplicate',
'changelog-producer' = 'input',
"sink.parallelism" = "4"
);

The table is created as above. When a large amount of data is written or a large amount of data is changed, the following error will appear. However, when sink.parallelism is 1, it will not appear. My data is stored on S3 and uses hivemetastore 3.1.2.Is there something wrong with my usage? Is there any way to increase the write parallelism?

Image

### What doesn't meet your expectations?

I hope that even when the data is stored on S3, which does not have atomic storage, it can still support multiple concurrent writes.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the CREATE TABLE and write workload with sink.parallelism set to 1 and 4 on S3, using Flink 1.20.1 and Hive Metastore 3.1.2, and capture the full Hive-lock error shown in the attachment. Trace the concurrent-write and locking path to determine whether parallel writes are supported; done means the failure has a confirmed cause and a verified fix or documented limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.