[Umbrella] Kyuubi Spark Kudu Connector
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues) and found no similar issues.
### Describe the proposal
https://github.com/apache/incubator-kyuubi/issues/2538 introduce a DSv2-based TPC-DS connector which make user use Spark SQL/Dataframe to access external data as Spark tables, this umbrella proposes to implement a DSv2-based connector for Apache Kudu.
Limitation: DSv2 API is not fully compatible across Spark minor versions, e.g. Spark 3.2 & 3.3. We are aim to implement on Spark 3.3 firstly.
### Task list
- https://github.com/apache/incubator-kyuubi/issues/2602
- Introduce new module and setup testcontainers-based Kudu service for testing
- Initial implement a DSv2-based Kudu connector
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Contributor guide
Research direction
Start by reading linked issue #2602 and the DSv2 connector proposal in issue #2538. Then identify how a new module and testcontainers-based Kudu service fit the Spark 3.3 build. Done means a tested initial Apache Kudu connector that exposes Kudu data through Spark SQL/DataFrame tables.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100