apache / apache/kyuubi

[Umbrella] Kyuubi Spark Kudu Connector

Open
#2,577 13 comments 3 reactions 0 assignees View on GitHub
kind:umbrella priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues) and found no similar issues.

### Describe the proposal

https://github.com/apache/incubator-kyuubi/issues/2538 introduce a DSv2-based TPC-DS connector which make user use Spark SQL/Dataframe to access external data as Spark tables, this umbrella proposes to implement a DSv2-based connector for Apache Kudu.

Limitation: DSv2 API is not fully compatible across Spark minor versions, e.g. Spark 3.2 & 3.3. We are aim to implement on Spark 3.3 firstly.

### Task list

- https://github.com/apache/incubator-kyuubi/issues/2602
- Introduce new module and setup testcontainers-based Kudu service for testing
- Initial implement a DSv2-based Kudu connector

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reading linked issue #2602 and the DSv2 connector proposal in issue #2538. Then identify how a new module and testcontainers-based Kudu service fit the Spark 3.3 build. Done means a tested initial Apache Kudu connector that exposes Kudu data through Spark SQL/DataFrame tables.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.