[Improvement] Kyuubi pool-share-level current policies can not dynamic expand or decrease engin according to engin load
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
Add a new rebalanced policy to increase or decrease engin according to current jdbc connections.
### How should we improve?
Because one engine for one namespace , We can limit the number of connections per engine by adding engin expand threshold. When a new connection come, we choose spaces whose conn num is less than the expand threshold, then we choose the suitable space with the highest conn num from choosed spaces . The purpose of this is to place the jdbc connection on the engine with high load as much as possible, and the engine with low load will automatically starve to death after the timeout, so as to achieve the purpose of reducing the engine.
### Are you willing to submit PR?
- [ ] Yes. I can submit a PR independently to improve.
- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.
Contributor guide
Research direction
Start by reading the current pool-share-level policy and engine-selection entry points, then trace how current JDBC connection counts and engine timeouts are handled. Define the rebalanced policy around an engine expansion threshold, selecting a namespace with the highest connection count below that threshold. Done means engines can expand or shrink dynamically as connection load changes, with low-load engines eventually timing out.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100