apache / apache/iggy

Implement Databricks sink connector

Open
#3,469 2 comments 0 reactions 1 assignee Claimed by @mmmmxa View on GitHub
connectors
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

### Description

Add a Databricks connector for Apache Iggy.

Recommendation: scope the first connector as a **sink**, not a combined sink + source. Databricks is primarily a lakehouse/warehouse destination for streaming and event pipelines, and Iggy's connector model maps that naturally to `Iggy stream -> sink -> Databricks table`.

A Databricks source can be useful later via Delta Change Data Feed (`table_changes`) or SQL polling, but it has a different stateful contract: table-version cursors, chunk/page offsets, CDF retention windows, and schema-change behavior. Keeping the source separate should make implementation, review, and testing more tractable.

Related context:

- #2753 lists Databricks as a proposed data warehouse **sink** target.
- #1852 tracks Delta Lake connectors, and `core/connectors/sinks/delta_sink` already covers direct Delta table/object-store writes. A Databricks connector should cover Databricks workspace, SQL warehouse, Unity Catalog, and managed-table ingestion flows that are not addressed by a raw Delta table URI alone.

References:

- Databricks SQL Statement Execution API: https://docs.databricks.com/aws/en/dev-tools/sql-execution-tutorial
- Databricks `COPY INTO`: https://docs.databricks.com/aws/en/sql/language-manual/delta-copy-into
- Databricks Change Data Feed for a possible future source connector: https://docs.databricks.com/aws/en/tables/features/change-data-feed

### Affected area / component

Connectors

### Proposed solution

Add `core/connectors/sinks/databricks_sink/` implementing `iggy_connector_sdk::Sink`.

Suggested scope:

- Config fields for workspace host, authentication token/OAuth/service-principal credentials, SQL warehouse ID, catalog/schema/table, ingestion mode, batch size, timeout/retry options, and optional staging location or Unity Catalog volume.
- Store all credentials as `SecretString` and ensure serialized/debug output redacts secrets.
- MVP ingestion path: batch Iggy messages and load them into a Databricks table through the Databricks SQL Statement Execution API against a SQL warehouse.
- Preferred high-throughput path: stage JSON or Parquet files to cloud storage or a Unity Catalog volume, then run `COPY INTO` or `MERGE` through the Statement Execution API. This should fit retry/replay behavior better than per-row inserts because `COPY INTO` can skip already-loaded files.
- Follow existing connector patterns from `delta_sink`, `iceberg_sink`, and `doris_sink` for batching, retryable vs permanent errors, logging, metrics, secret handling, docs, and example config.
- Add focused tests for config validation, SQL/request generation, retry/error mapping, and docs/examples. Integration testing may need to be gated behind Databricks credentials or covered with a local HTTP mock first.

### Contribution

- [ ] I'm willing to submit a pull request to implement this feature

### Good first issue

- [ ] I think this could be a good first issue for a new contributor

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.