destination-databricks: Add CLUSTER BY AUTO option for table creation
- Ngôn ngữ chính
- Python
- Star
- 22.1k
- Fork
- 5.3k
- Chỉ số merge pull request
- Chỉ số pull request đang chờ
Mô tả
## Summary
Add a configuration option to the Databricks destination connector that enables `CLUSTER BY AUTO` on created tables. This feature allows Databricks to automatically optimize table layout for query performance, reducing full table scans when running transformation jobs that pull data from source tables.
## Motivation
When building transformation jobs that pull data from Airbyte-synced tables, users often need to perform queries that would benefit from clustered tables. Without clustering, these transformation jobs may perform full table scans each time they run, which is inefficient and costly.
Databricks' `CLUSTER BY AUTO` feature automatically determines the optimal clustering columns based on query patterns, making it an ideal default optimization for tables that will be queried by downstream transformation jobs.
## Proposed Implementation
1. Add a new boolean config option `enable_auto_clustering` (default: `false` for backward compatibility) to `spec.json`
2. Update `DatabricksConnectorConfig.kt` to deserialize the new config field
3. Modify `DatabricksSqlGenerator.createTable()` to append `CLUSTER BY AUTO` when the option is enabled
4. Optionally also update `createRawTable()` if raw tables should also be clustered
## References
- [Databricks CLUSTER BY AUTO documentation](https://docs.databricks.com/en/delta/clustering.html)
- Current table creation code: `airbyte-integrations/connectors/destination-databricks/src/main/kotlin/io/airbyte/integrations/destination/databricks/jdbc/DatabricksSqlGenerator.kt`
## Requester
This feature was requested by @iherdt-airbyte via Slack (#ask-devin-ai channel).
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.