dbt-labs / dbt-labs/dbt-adapters
[CT-975] [Feature] Parity on `create_table_as` for python and SQL model
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
## Describe the feature
[spark__create_table_as](https://github.com/dbt-labs/dbt-spark/blob/f58fc233fbbc7b9289299313807e706c57360613/dbt/include/spark/macros/adapters.sql#L143) macro support options including `partition_by`, `clustered_by`, `file_format`, `location_root`, and more options defined in [options_clause](https://github.com/dbt-labs/dbt-spark/blob/f58fc233fbbc7b9289299313807e706c57360613/dbt/include/spark/macros/adapters.sql#L30).
Right now in python models we are just [saving everything as delta format](https://github.com/dbt-labs/dbt-spark/blob/main/dbt/include/spark/macros/materializations/table.sql#L44) with the default setting. We should reach parity for this for python models where possible and raise a clear error when running with options that is not supported.
## Motivation:
User would be able to optimize the storage format based on their usage of the table.
## Acceptance criteria
Python model would materialize the table with the correct option, and raise error when unsupported option is being specified.
## Tests for the PR
You should add integration tests to run the table materialization with supported options, then check that the table has intended property, for example `SHOW PARTITION table`([link](https://docs.databricks.com/spark/latest/spark-sql/language-manual/sql-ref-syntax-aux-show-partitions.html)) can be used to check partitions. You should also add tests to to make sure we raised the error on unsupported options.
Contributor guide
Assessment
This issue has not been assessed yet.