apache / apache/kyuubi

[FEATURE] 让 Kyuubi Engine 跑在阿里MaxCompute或AWS Glue上

Open
#3,409 18 comments 0 reactions 0 assignees View on GitHub
kind:feature priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the feature

目前Kyuubi Engine可以运行在Yarn或K8s上以执行通过JDBC提交的任务,但在云原生环境里,通常云提供商都提供弹性的云计算资源,比如阿里云的MaxCompute和AWS Glue。如果Kyuubi Engine支持运行在MaxCompute和Glue上,可以大大降低Spark的运行成本和维护成本。

阿里云的通过MaxCompute运行spark任务的API:
https://help.aliyun.com/document_detail/102357.html

AWS的通过Glue运行spark任务的API:
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-jobs-job.html#aws-glue-api-jobs-job-CreateJob

### Motivation

目前Kyuubi Engine只能运行在Yarn或K8S上,这样在云原生的环境里要么需要申请EMR资源,要么需要申请K8S计算节点,这里存在两个问题:
1. EMR和K8S的资源不是弹性的,当任务少时,不能缩容以减少硬件成本,当任务多时,不能扩容,以提高计算速度。
2. 在云环境中,如果使用MaxCompute这样的弹性计算资源,JDBC只能使用Trino这样的交互式查询引擎,造成离线任务和交互式查询的SQL标准不完全一致。

### Describe the solution

通过将Kyuubi Engine运行在MaxCompute和Glue这种弹性Spark计算资源上,可以让离线批量任务和交互式查询共用相同的spark sql能力,也可以让计算资源有弹性,节省基础设施成本和运维成本。

### Additional context

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reviewing how Kyuubi Engine currently runs on YARN and Kubernetes, then read the referenced MaxCompute and AWS Glue job APIs. Done would mean defining and implementing engine execution on both elastic Spark services for JDBC-submitted tasks, with the required integration behavior and validation identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes, scala, spark
Domain
backend, cloud, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.