airbytehq / airbytehq/PyAirbyte
Add a Databricks SQL cache to PyAirbyte
- Lenguaje dominante
- Python
- Estrellas
- 344
- Forks
- 77
- Merge medio
- 1 d 11 h
- PR fusionados (30 d)
- 35
Descripción
## Summary
Add a native Databricks SQL cache implementation to PyAirbyte (`airbyte.caches.databricks`), alongside the existing DuckDB, MotherDuck, Postgres, Snowflake, and BigQuery caches. It would pair with the existing `destination-databricks` (Delta Lake on Unity Catalog) the same way `SnowflakeCache` and `BigQueryCache` pair with their destination connectors.
## Motivation
Code-first data engineering teams want to run an entire pipeline inside their own Python/Airflow stack: extract with PyAirbyte, transform with dbt, and land in Databricks, all in one repo and one runtime, without bouncing between an orchestrator, a separate sync service, and a connector UI.
Today PyAirbyte can write to Databricks only via the `destination-databricks` connector through `Destination.write()`. Because that destination is Java-based, it runs as a Docker image, so the path requires Docker on every Airflow worker. That breaks the "pure Python, no containers" model that makes the SQL caches attractive in the first place.
A native Databricks cache, implemented in-process like the other SQL caches, would:
- Remove the Docker dependency for Databricks-bound pipelines.
- Let `cache` and final destination be the same system, instead of using a throwaway DuckDB buffer.
- Give teams a single, portable, Python-native end-to-end flow (read -> cache in Databricks -> dbt -> done) runnable anywhere PyAirbyte runs.
## Proposed solution
- Add `DatabricksCache` under `airbyte.caches.databricks`, following the existing `CacheBase` / `SqlConfig` pattern used by `SnowflakeCache` and `BigQueryCache`.
- Back it with a SQLAlchemy-compatible Databricks dialect (e.g. `databricks-sql-connector` / `databricks-sqlalchemy`) against a SQL warehouse / Unity Catalog.
- Set `paired_destination_name` / `paired_destination_config_class` to `destination-databricks` so `clone_as_cloud_destination_config()` and destination pairing work consistently with the other caches.
- Support the usual cache responsibilities: stream catalog, state tracking, and read/write through the SQL backend.
## Alternatives considered
- **`Destination.write()` with the Databricks destination connector** works today but requires Docker on the runtime, which defeats the Python-native goal.
- **DuckDB/Postgres/Snowflake/BigQuery cache as an intermediate buffer, then load to Databricks** adds a hop and still needs the Docker-based destination for the final load.
## Additional context
An enterprise design partner with a Databricks + Airflow + dbt stack has offered to help test an alpha and work with us toward release. Happy to connect the maintainers with them to validate the design and shake out auth (token vs OAuth) and Unity Catalog specifics.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.