apache / apache/kyuubi

[FEATURE] Make external KV store as an option to store session and operation handles

Open
#1,754 11 comments 13 reactions 0 assignees View on GitHub
kind:feature
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the feature

Currently, we use in-memory maps to store session and operation handles, is it possible to store them via leveldb, zookeeper, redis? Then we can make kyuubi server free to rebooting as the actual executions are on the engine side

### Motivation

make kyuubi server stateless

### Describe the solution

_No response_

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points; start by locating the in-memory session and operation-handle maps. Define how an external store such as Redis would be selected and how handles survive a Kyuubi server restart, then verify that the server can reboot while executions continue on the engine side.

Written by the indexing model from the issue text.

Assessment

Tech stack
redis, scala
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.