apache / apache/kyuubi

[Improvement] Python Process Pooling in Python Mode

Open
#6,637 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### What would you like to be improved?

In Python mode, the first execution of each session is relatively slow, and after investigation, it was found that the slow creation of Python processes is the cause

image

### How should we improve?

Design a Python process pool, pre create Python processes to avoid the creation of Python processes affecting code execution speed

### Are you willing to submit PR?

- [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by locating the Python mode process-creation path and measuring the first session execution; define the pool lifecycle and verify that pre-created processes reduce the startup delay without changing session behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.