airbytehq / airbytehq/airbyte

destination-databricks: Add CLUSTER BY AUTO option for table creation

未關閉
#70,993 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
autoteam connectors/destination/databricks team/destinations
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

## Summary

Add a configuration option to the Databricks destination connector that enables `CLUSTER BY AUTO` on created tables. This feature allows Databricks to automatically optimize table layout for query performance, reducing full table scans when running transformation jobs that pull data from source tables.

## Motivation

When building transformation jobs that pull data from Airbyte-synced tables, users often need to perform queries that would benefit from clustered tables. Without clustering, these transformation jobs may perform full table scans each time they run, which is inefficient and costly.

Databricks' `CLUSTER BY AUTO` feature automatically determines the optimal clustering columns based on query patterns, making it an ideal default optimization for tables that will be queried by downstream transformation jobs.

## Proposed Implementation

1. Add a new boolean config option `enable_auto_clustering` (default: `false` for backward compatibility) to `spec.json`
2. Update `DatabricksConnectorConfig.kt` to deserialize the new config field
3. Modify `DatabricksSqlGenerator.createTable()` to append `CLUSTER BY AUTO` when the option is enabled
4. Optionally also update `createRawTable()` if raw tables should also be clustered

## References

- [Databricks CLUSTER BY AUTO documentation](https://docs.databricks.com/en/delta/clustering.html)
- Current table creation code: `airbyte-integrations/connectors/destination-databricks/src/main/kotlin/io/airbyte/integrations/destination/databricks/jdbc/DatabricksSqlGenerator.kt`

## Requester

This feature was requested by @iherdt-airbyte via Slack (#ask-devin-ai channel).

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。