dbt-labs / dbt-labs/dbt-adapters

[CT-975] [Feature] Parity on `create_table_as` for python and SQL model

Open
#511 7 comments 2 reactions 0 assignees View on GitHub
feature:python-models pkg:dbt-spark type:enhancement
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

## Describe the feature
[spark__create_table_as](https://github.com/dbt-labs/dbt-spark/blob/f58fc233fbbc7b9289299313807e706c57360613/dbt/include/spark/macros/adapters.sql#L143) macro support options including `partition_by`, `clustered_by`, `file_format`, `location_root`, and more options defined in [options_clause](https://github.com/dbt-labs/dbt-spark/blob/f58fc233fbbc7b9289299313807e706c57360613/dbt/include/spark/macros/adapters.sql#L30).

Right now in python models we are just [saving everything as delta format](https://github.com/dbt-labs/dbt-spark/blob/main/dbt/include/spark/macros/materializations/table.sql#L44) with the default setting. We should reach parity for this for python models where possible and raise a clear error when running with options that is not supported.

## Motivation:
User would be able to optimize the storage format based on their usage of the table.

## Acceptance criteria
Python model would materialize the table with the correct option, and raise error when unsupported option is being specified.

## Tests for the PR
You should add integration tests to run the table materialization with supported options, then check that the table has intended property, for example `SHOW PARTITION table`([link](https://docs.databricks.com/spark/latest/spark-sql/language-manual/sql-ref-syntax-aux-show-partitions.html)) can be used to check partitions. You should also add tests to to make sure we raised the error on unsupported options.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.