apache / apache/doris-flink-connector
[Enhancement]
- Dominant language
- Java
- Stars
- 385
- Forks
- 279
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 6
Description
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
In version 1.4.0,Release Note 1.4.0 shows:
6. Version 1.4.0 introduces a polling mechanism, and BE nodes are replaced during each Checkpoint, avoiding the pressure of a single node acting as a Coordinator for a long time.
The Improvement is friendly to one stream load task.
but In a high concurrency scenario, the load task is distributed to a BE node, resulting in great pressure on this node. my environment 72 parallelism, 3 FE node, 6 BE node, one BE node as 72 Coordinator is too heavy just some time after startup。traffic needs to be distributed to all BE nodes in a near-average.
### Solution
In DorisWriter.initializeLoad() 107 line.
//cache backend
this.backends = RestService.getBackendsV2(dorisOptions, dorisReadOptions, LOG);
Collections.shuffle(backends); //Is it possible to add this line ?
String backend = getAvailableBackend();
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start in DorisWriter.initializeLoad() around line 107 and inspect how the cached backends from RestService.getBackendsV2 are used by getAvailableBackend(). Reproduce the reported high-concurrency behavior with multiple FE and BE nodes, then verify that load traffic is distributed near-evenly across available backends. Done means the selection no longer places all parallel tasks on one BE node.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100