apache / apache/kyuubi

[Improvement] Kyuubi pool-share-level current policies can not dynamic expand or decrease engin according to engin load

Open
#3,695 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

Add a new rebalanced policy to increase or decrease engin according to current jdbc connections.

### How should we improve?

Because one engine for one namespace , We can limit the number of connections per engine by adding engin expand threshold. When a new connection come, we choose spaces whose conn num is less than the expand threshold, then we choose the suitable space with the highest conn num from choosed spaces . The purpose of this is to place the jdbc connection on the engine with high load as much as possible, and the engine with low load will automatically starve to death after the timeout, so as to achieve the purpose of reducing the engine.

### Are you willing to submit PR?

- [ ] Yes. I can submit a PR independently to improve.
- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start by reading the current pool-share-level policy and engine-selection entry points, then trace how current JDBC connection counts and engine timeouts are handled. Define the rebalanced policy around an engine expansion threshold, selecting a namespace with the highest connection count below that threshold. Done means engines can expand or shrink dynamically as connection load changes, with low-load engines eventually timing out.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.