[Bug] flinksql two sinks can't define parallelism
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
when I created a FLINKSQL script in dinky,the source data was in paimon,and the table buckets were 32,there were two sinks,one was to paimon,another was to kafka,I setted all the scan parallelism = 1 and sink.parallelism = 1,however,when I started the script,it createed more then 300 operators,which exhausts all my flink slots and made the performance failed .
### Compute Engine
Flink1.18
### Minimal reproduce step
create flinksql script,createtwo sinks,one is to paimon,one is to kafka
### What doesn't meet your expectations?
I expected one parallelism,but it takes 300+ parallelisms
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the reported Flink 1.18 SQL reproduction: a Paimon source with 32 buckets and two sinks, one to Paimon and one to Kafka, while scan and sink parallelism are set to 1. Inspect the generated operators and parallelism settings; done means the two-sink job respects the requested parallelism without exhausting the available Flink slots.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100