apache / apache/kyuubi

[Umbrella] Kyuubi Spark TPC-DS Connector

Open
#2,538 4 comments 2 reactions 0 assignees View on GitHub
good first issue help wanted kind:feature kind:umbrella module:spark priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [X] I have searched in the [issues](https://github.com/apache/incubator-kyuubi/issues) and found no similar issues.

### Describe the proposal

Spark DataSource V2 API[1] is available since Spark 3.0, basically, it provides a bunch of APIs for developers to implement a connector, and Spark will expose them to SQL/DataFrame API automatically with few configurations.

TPC-DS[2] dataset is very useful for benchmarking and demonstration. Previously, we need to generate the dataset by using `dsdgen` or `kyuubi-tpcds` tool before running queries. With the connector proposed by this PR, users just need

1. Add jar `kyuubi-spark-connector-tpcds_2.12-${kyuubi_version}.jar`
2. Add conf `spark.sql.catalog.tpcds=org.apache.kyuubi.spark.connector.tpcds.TPCDSCatalog`

Then they can query the different scales of TPC-DS tables under `tpcds.sf{scale}` database. For instance,

```
0: jdbc:hive2://0.0.0.0:10009/> show tables in tpcds.sf1;
+------------+-------------------------+--------------+
| namespace | tableName | isTemporary |
+------------+-------------------------+--------------+
| sf1 | call_center | false |
| sf1 | catalog_page | false |
| sf1 | catalog_returns | false |
| sf1 | catalog_sales | false |
| sf1 | customer | false |
| sf1 | customer_address | false |
| sf1 | customer_demographics | false |
| sf1 | date_dim | false |
| sf1 | household_demographics | false |
| sf1 | income_band | false |
| sf1 | inventory | false |
| sf1 | item | false |
| sf1 | promotion | false |
| sf1 | reason | false |
| sf1 | ship_mode | false |
| sf1 | store | false |
| sf1 | store_returns | false |
| sf1 | store_sales | false |
| sf1 | time_dim | false |
| sf1 | warehouse | false |
| sf1 | web_page | false |
| sf1 | web_returns | false |
| sf1 | web_sales | false |
| sf1 | web_site | false |
+------------+-------------------------+--------------+
```

[1] https://github.com/apache/spark/tree/v3.2.1/sql/catalyst/src/main/java/org/apache/spark/sql/connector
[2] https://tpc.org/TPC_Documents_Current_Versions/pdf/TPC-DS_v3.2.0.pdf

### Task list

- https://github.com/apache/incubator-kyuubi/pull/2531
- https://github.com/apache/incubator-kyuubi/issues/2539
- https://github.com/apache/incubator-kyuubi/issues/2540
- https://github.com/apache/incubator-kyuubi/issues/2541
- https://github.com/apache/incubator-kyuubi/issues/2542
- https://github.com/apache/incubator-kyuubi/issues/2543
- https://github.com/apache/incubator-kyuubi/issues/2553
- https://github.com/apache/incubator-kyuubi/pull/2673
- https://github.com/apache/incubator-kyuubi/issues/2679
- https://github.com/apache/incubator-kyuubi/pull/2700
- https://github.com/apache/incubator-kyuubi/pull/2701
- https://github.com/apache/incubator-kyuubi/pull/2702
- https://github.com/apache/incubator-kyuubi/issues/2704
- https://github.com/apache/incubator-kyuubi/pull/2709
- https://github.com/apache/incubator-kyuubi/pull/2729
- https://github.com/apache/incubator-kyuubi/issues/2759
- https://github.com/apache/incubator-kyuubi/pull/2777

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Begin with linked PR #2531 and the listed task issues and pull requests, then review the Spark DataSource V2 and TPC-DS references in the proposal. The umbrella work is complete when the connector can be packaged, configured as shown, and used to query the listed TPC-DS tables at different scales.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.