airbytehq / airbytehq/PyAirbyte

Add a Databricks SQL cache to PyAirbyte

Đang mở
#1,047 2 bình luận 2 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
344
Fork
77
Merge trung bình
1 ngày 11 giờ
Pull request đã merge (30 ngày)
35

Mô tả

## Summary

Add a native Databricks SQL cache implementation to PyAirbyte (`airbyte.caches.databricks`), alongside the existing DuckDB, MotherDuck, Postgres, Snowflake, and BigQuery caches. It would pair with the existing `destination-databricks` (Delta Lake on Unity Catalog) the same way `SnowflakeCache` and `BigQueryCache` pair with their destination connectors.

## Motivation

Code-first data engineering teams want to run an entire pipeline inside their own Python/Airflow stack: extract with PyAirbyte, transform with dbt, and land in Databricks, all in one repo and one runtime, without bouncing between an orchestrator, a separate sync service, and a connector UI.

Today PyAirbyte can write to Databricks only via the `destination-databricks` connector through `Destination.write()`. Because that destination is Java-based, it runs as a Docker image, so the path requires Docker on every Airflow worker. That breaks the "pure Python, no containers" model that makes the SQL caches attractive in the first place.

A native Databricks cache, implemented in-process like the other SQL caches, would:

- Remove the Docker dependency for Databricks-bound pipelines.
- Let `cache` and final destination be the same system, instead of using a throwaway DuckDB buffer.
- Give teams a single, portable, Python-native end-to-end flow (read -> cache in Databricks -> dbt -> done) runnable anywhere PyAirbyte runs.

## Proposed solution

- Add `DatabricksCache` under `airbyte.caches.databricks`, following the existing `CacheBase` / `SqlConfig` pattern used by `SnowflakeCache` and `BigQueryCache`.
- Back it with a SQLAlchemy-compatible Databricks dialect (e.g. `databricks-sql-connector` / `databricks-sqlalchemy`) against a SQL warehouse / Unity Catalog.
- Set `paired_destination_name` / `paired_destination_config_class` to `destination-databricks` so `clone_as_cloud_destination_config()` and destination pairing work consistently with the other caches.
- Support the usual cache responsibilities: stream catalog, state tracking, and read/write through the SQL backend.

## Alternatives considered

- **`Destination.write()` with the Databricks destination connector** works today but requires Docker on the runtime, which defeats the Python-native goal.
- **DuckDB/Postgres/Snowflake/BigQuery cache as an intermediate buffer, then load to Databricks** adds a hop and still needs the Docker-based destination for the final load.

## Additional context

An enterprise design partner with a Databricks + Airflow + dbt stack has offered to help test an alpha and work with us toward release. Happy to connect the maintainers with them to validate the design and shake out auth (token vs OAuth) and Unity Catalog specifics.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.