apache / apache/pulsar

To optimize parallelism config in config file

Open
#14,281 2 comments 0 reactions 0 assignees View on GitHub
help wanted lifecycle/stale Stale type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Is your enhancement request related to a problem? Please describe.**
in config file of pulsar io ,
user can define the parallelism parameter,
but there is a problem, for example
In Mongodb Source, if this param is set to be 100000,
when run in local run mode,100000 threads generated, but apparently this is wrong,
how about the config set to be 10000000000?
maybe u can say, this is set by user ,but we can build pulsar io stronger as posssible to reduce errors
because users do not know more details , and they are not expected to be expert of all Storages.

besides this, in Mongo Source ,the parallelism just shoud be 1, if above 1, the result will be wrong

so, based on the 2 reasons, the parallelism config should be decided finally by the Source Class , just like kafka connect
because the author of the Source Class knows concretely how the source runs

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

Start by tracing how Pulsar IO reads the parallelism setting and how the MongoDB Source uses it in local run mode. Compare the requested source-specific control with the stated Kafka Connect behavior, then determine safe limits and validation rules. Done means excessive values cannot create unbounded threads and MongoDB Source parallelism cannot produce incorrect results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mongodb
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.