apache / apache/paimon

[Bug] flinksql two sinks can't define parallelism

Open
#5,635 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

when I created a FLINKSQL script in dinky,the source data was in paimon,and the table buckets were 32,there were two sinks,one was to paimon,another was to kafka,I setted all the scan parallelism = 1 and sink.parallelism = 1,however,when I started the script,it createed more then 300 operators,which exhausts all my flink slots and made the performance failed .

### Compute Engine

Flink1.18

### Minimal reproduce step

create flinksql script,createtwo sinks,one is to paimon,one is to kafka

### What doesn't meet your expectations?

I expected one parallelism,but it takes 300+ parallelisms

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reported Flink 1.18 SQL reproduction: a Paimon source with 32 buckets and two sinks, one to Paimon and one to Kafka, while scan and sink parallelism are set to 1. Inspect the generated operators and parallelism settings; done means the two-sink job respects the requested parallelism without exhausting the available Flink slots.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.