dbt-labs / dbt-labs/dbt-adapters

[Feature] Support OCI Dataflow as a backend for dbt-spark

Open
#484 2 comments 0 reactions 0 assignees View on GitHub
pkg:dbt-spark type:enhancement
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt-spark functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

OCI Dataflow is an Oracle serverless Spark solution that also has a SQL Endpoint accessible via Simba ODBC driver for example. The Simba OCI driver is similar to the Databricks one but the properties are slightly different. One of the changes is that sparkServerType property passed in the connection must have value DFI - in the code here it is hard-wired to the value 3. The odbc code path also fails if cluster is not set or endpoint is not set, these are not needed for the OCI Simba ODBC driver (along with validation failure).
if creds.cluster is not None:
...
elif creds.endpoint is not None:
...
else:
...raises error

A working format for the Simba OCI ODBC driver has the JDBC URL as; jdbc:spark://yourendpoint.oci.oraclecloud.com/default;SparkServerType=DFI;httpPath=cliservice

I have made my own local modifications to the connections.py code (in adapters/spark/connections.py) and been able to use Delta format in OCI Dataflow SQL endpoint to do incremental extracts and merges for example. See here;
https://github.com/davidallan/dbt-spark/blob/main/dbt/adapters/spark/connections.py

### Describe alternatives you've considered

_No response_

### Who will this benefit?

This will open up DBT to the OCI serverless spark platform.

### Are you interested in contributing this feature?

Yes, interested in doing this.

### Anything else?

Here is the doc on OCI SQL Endpoints for OCI Dataflow;
https://docs.oracle.com/en-us/iaas/data-flow/using/sql-endpoints-connect.htm

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.