airbytehq / airbytehq/airbyte

destination-databricks: Add CLUSTER BY AUTO option for table creation

Đang mở
#70,993 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
autoteam connectors/destination/databricks team/destinations
Ngôn ngữ chính
Python
Star
22.1k
Fork
5.3k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

## Summary

Add a configuration option to the Databricks destination connector that enables `CLUSTER BY AUTO` on created tables. This feature allows Databricks to automatically optimize table layout for query performance, reducing full table scans when running transformation jobs that pull data from source tables.

## Motivation

When building transformation jobs that pull data from Airbyte-synced tables, users often need to perform queries that would benefit from clustered tables. Without clustering, these transformation jobs may perform full table scans each time they run, which is inefficient and costly.

Databricks' `CLUSTER BY AUTO` feature automatically determines the optimal clustering columns based on query patterns, making it an ideal default optimization for tables that will be queried by downstream transformation jobs.

## Proposed Implementation

1. Add a new boolean config option `enable_auto_clustering` (default: `false` for backward compatibility) to `spec.json`
2. Update `DatabricksConnectorConfig.kt` to deserialize the new config field
3. Modify `DatabricksSqlGenerator.createTable()` to append `CLUSTER BY AUTO` when the option is enabled
4. Optionally also update `createRawTable()` if raw tables should also be clustered

## References

- [Databricks CLUSTER BY AUTO documentation](https://docs.databricks.com/en/delta/clustering.html)
- Current table creation code: `airbyte-integrations/connectors/destination-databricks/src/main/kotlin/io/airbyte/integrations/destination/databricks/jdbc/DatabricksSqlGenerator.kt`

## Requester

This feature was requested by @iherdt-airbyte via Slack (#ask-devin-ai channel).

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.