To optimize parallelism config in config file
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Is your enhancement request related to a problem? Please describe.**
in config file of pulsar io ,
user can define the parallelism parameter,
but there is a problem, for example
In Mongodb Source, if this param is set to be 100000,
when run in local run mode,100000 threads generated, but apparently this is wrong,
how about the config set to be 10000000000?
maybe u can say, this is set by user ,but we can build pulsar io stronger as posssible to reduce errors
because users do not know more details , and they are not expected to be expert of all Storages.
besides this, in Mongo Source ,the parallelism just shoud be 1, if above 1, the result will be wrong
so, based on the 2 reasons, the parallelism config should be decided finally by the Source Class , just like kafka connect
because the author of the Source Class knows concretely how the source runs
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
Start by tracing how Pulsar IO reads the parallelism setting and how the MongoDB Source uses it in local run mode. Compare the requested source-specific control with the stated Kafka Connect behavior, then determine safe limits and validation rules. Done means excessive values cannot create unbounded threads and MongoDB Source parallelism cannot produce incorrect results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mongodb
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100