apache / apache/drill

Intelligent client connection load balancing

Open
#2,978 3 comments 0 reactions 1 assignee Claimed by @rymarm View on GitHub
Dominant language
Java
Stars
2k
Forks
990
Avg merge
1d 8h
Merged PRs (30d)
5

Description

One drillbit may end up with significantly more client connections than others.

Here is an example of such a case:
![Image](https://github.com/user-attachments/assets/b4a1c035-0e00-4258-b9ff-76efd7f2ce10)

The chart shows that the highest number of connections per node is 21, while the lowest is only 3 – a difference of 7 times.

Even though query execution considers the load on all Drillbits and distributes execution accordingly, overutilizing a single Drillbit for client connections can lead to issues such as heap memory exhaustion and excessive network bandwidth usage when returning query results to the client.

The current client implementation retrieves a list of active Drillbits from Zookeeper, randomly shuffles them:
```java
java.util.Collections.shuffle(endpoints);
```
and attempts to connect to the first N nodes until a connection is successful. This approach was introduced as a simple way to prevent all clients from connecting to the same Drillbit: https://issues.apache.org/jira/browse/DRILL-2512. However, it does not fully solve the problem, as one node may still end up with significantly more connections than others.

**Solution**
Every drillbit stores the his current number of client connections in Zookeeper. After retrieving a list of active drillbits, the client calculates the average number of connections and reorders the list based on whether a drillbit's connection count is above or below the average before attempting a connection.

The key questions I see here are:
* Is it appropriate to store information about the current number of user connections in Zookeeper?
* Should client connection load balancing be enabled be configurable and enabled by default?
* Is it beneficial to distribute all connections evenly across the entire Drill cluster?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.