airbytehq / airbytehq/PyAirbyte

Add a Databricks SQL cache to PyAirbyte

未关闭
#1,047 2 条评论 2 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
344
派生
77
平均合并
1 天 11 小时
30 天内合并 PR
35

描述

## Summary

Add a native Databricks SQL cache implementation to PyAirbyte (`airbyte.caches.databricks`), alongside the existing DuckDB, MotherDuck, Postgres, Snowflake, and BigQuery caches. It would pair with the existing `destination-databricks` (Delta Lake on Unity Catalog) the same way `SnowflakeCache` and `BigQueryCache` pair with their destination connectors.

## Motivation

Code-first data engineering teams want to run an entire pipeline inside their own Python/Airflow stack: extract with PyAirbyte, transform with dbt, and land in Databricks, all in one repo and one runtime, without bouncing between an orchestrator, a separate sync service, and a connector UI.

Today PyAirbyte can write to Databricks only via the `destination-databricks` connector through `Destination.write()`. Because that destination is Java-based, it runs as a Docker image, so the path requires Docker on every Airflow worker. That breaks the "pure Python, no containers" model that makes the SQL caches attractive in the first place.

A native Databricks cache, implemented in-process like the other SQL caches, would:

- Remove the Docker dependency for Databricks-bound pipelines.
- Let `cache` and final destination be the same system, instead of using a throwaway DuckDB buffer.
- Give teams a single, portable, Python-native end-to-end flow (read -> cache in Databricks -> dbt -> done) runnable anywhere PyAirbyte runs.

## Proposed solution

- Add `DatabricksCache` under `airbyte.caches.databricks`, following the existing `CacheBase` / `SqlConfig` pattern used by `SnowflakeCache` and `BigQueryCache`.
- Back it with a SQLAlchemy-compatible Databricks dialect (e.g. `databricks-sql-connector` / `databricks-sqlalchemy`) against a SQL warehouse / Unity Catalog.
- Set `paired_destination_name` / `paired_destination_config_class` to `destination-databricks` so `clone_as_cloud_destination_config()` and destination pairing work consistently with the other caches.
- Support the usual cache responsibilities: stream catalog, state tracking, and read/write through the SQL backend.

## Alternatives considered

- **`Destination.write()` with the Databricks destination connector** works today but requires Docker on the runtime, which defeats the Python-native goal.
- **DuckDB/Postgres/Snowflake/BigQuery cache as an intermediate buffer, then load to Databricks** adds a hop and still needs the Docker-based destination for the final load.

## Additional context

An enterprise design partner with a Databricks + Airflow + dbt stack has offered to help test an alpha and work with us toward release. Happy to connect the maintainers with them to validate the design and shake out auth (token vs OAuth) and Unity Catalog specifics.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。