Implement Databricks sink connector
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
### Description
Add a Databricks connector for Apache Iggy.
Recommendation: scope the first connector as a **sink**, not a combined sink + source. Databricks is primarily a lakehouse/warehouse destination for streaming and event pipelines, and Iggy's connector model maps that naturally to `Iggy stream -> sink -> Databricks table`.
A Databricks source can be useful later via Delta Change Data Feed (`table_changes`) or SQL polling, but it has a different stateful contract: table-version cursors, chunk/page offsets, CDF retention windows, and schema-change behavior. Keeping the source separate should make implementation, review, and testing more tractable.
Related context:
- #2753 lists Databricks as a proposed data warehouse **sink** target.
- #1852 tracks Delta Lake connectors, and `core/connectors/sinks/delta_sink` already covers direct Delta table/object-store writes. A Databricks connector should cover Databricks workspace, SQL warehouse, Unity Catalog, and managed-table ingestion flows that are not addressed by a raw Delta table URI alone.
References:
- Databricks SQL Statement Execution API: https://docs.databricks.com/aws/en/dev-tools/sql-execution-tutorial
- Databricks `COPY INTO`: https://docs.databricks.com/aws/en/sql/language-manual/delta-copy-into
- Databricks Change Data Feed for a possible future source connector: https://docs.databricks.com/aws/en/tables/features/change-data-feed
### Affected area / component
Connectors
### Proposed solution
Add `core/connectors/sinks/databricks_sink/` implementing `iggy_connector_sdk::Sink`.
Suggested scope:
- Config fields for workspace host, authentication token/OAuth/service-principal credentials, SQL warehouse ID, catalog/schema/table, ingestion mode, batch size, timeout/retry options, and optional staging location or Unity Catalog volume.
- Store all credentials as `SecretString` and ensure serialized/debug output redacts secrets.
- MVP ingestion path: batch Iggy messages and load them into a Databricks table through the Databricks SQL Statement Execution API against a SQL warehouse.
- Preferred high-throughput path: stage JSON or Parquet files to cloud storage or a Unity Catalog volume, then run `COPY INTO` or `MERGE` through the Statement Execution API. This should fit retry/replay behavior better than per-row inserts because `COPY INTO` can skip already-loaded files.
- Follow existing connector patterns from `delta_sink`, `iceberg_sink`, and `doris_sink` for batching, retryable vs permanent errors, logging, metrics, secret handling, docs, and example config.
- Add focused tests for config validation, SQL/request generation, retry/error mapping, and docs/examples. Integration testing may need to be gated behind Databricks credentials or covered with a local HTTP mock first.
### Contribution
- [ ] I'm willing to submit a pull request to implement this feature
### Good first issue
- [ ] I think this could be a good first issue for a new contributor
Contributor guide
Assessment
This issue has not been assessed yet.