apache / apache/kyuubi

[Improvement] enhancing TTL feature of spark engine instance

Open
#5,197 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

We are currently using the sharelevel+subdomain feature of kyuubi. It's a useful feature, and in most cases works well. only one problem is that some application will be long-living. Long-living application will have too many jobs , in our case the number of jobs will be 10000+. Considering the efficiency of reviewing history and stability, maybe this is not a good thing.

Now in kyuubi, Spark Engine Instance already has the concept of TTL, but it's more like IDLE_TIME_OUT.
Is this a good idea to enhance the TTL feature of Spark Engine Instance?

### How should we improve?

Add an optional TTL to spark instance, when some instance reaches its lifespan, a new rolling instance will be created to accept connections from the same subdomain.
two possible ways of TTL:
1) precise TTL
2)a time label, for example by nature day or hour

### Are you willing to submit PR?

- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

Start by locating the Spark engine instance TTL and IDLE_TIME_OUT implementation, then trace how sharelevel+subdomain instances accept connections. Compare precise TTL with hour- or day-based expiry and clarify the expected rolling-instance behavior with the community. Done means an agreed TTL design is implemented and validated for long-lived applications.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.