apache / apache/doris-flink-connector

[Enhancement]

Open
#158 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
385
Forks
279
Avg merge
2d 2h
Merged PRs (30d)
6

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.

### Description

In version 1.4.0,Release Note 1.4.0 shows:
6. Version 1.4.0 introduces a polling mechanism, and BE nodes are replaced during each Checkpoint, avoiding the pressure of a single node acting as a Coordinator for a long time.
The Improvement is friendly to one stream load task.
but In a high concurrency scenario, the load task is distributed to a BE node, resulting in great pressure on this node. my environment 72 parallelism, 3 FE node, 6 BE node, one BE node as 72 Coordinator is too heavy just some time after startup。traffic needs to be distributed to all BE nodes in a near-average.

### Solution

In DorisWriter.initializeLoad() 107 line.
//cache backend
this.backends = RestService.getBackendsV2(dorisOptions, dorisReadOptions, LOG);
Collections.shuffle(backends); //Is it possible to add this line ?
String backend = getAvailableBackend();

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start in DorisWriter.initializeLoad() around line 107 and inspect how the cached backends from RestService.getBackendsV2 are used by getAvailableBackend(). Reproduce the reported high-concurrency behavior with multiple FE and BE nodes, then verify that load traffic is distributed near-evenly across available backends. Done means the selection no longer places all parallel tasks on one BE node.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering
Issue type
Feature
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.