dbt-labs / dbt-labs/dbt-adapters

[CT-1842] [Feature] Support `on_schema_change='sync_all_columns'` for Delta tables

Open
#509 5 comments 3 reactions 0 assignees View on GitHub
help-wanted pkg:dbt-spark type:enhancement
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt-spark functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Delta Lake 2.0 supports dropping of columns on delta tables if the table has certain `tblproperties` (https://delta.io/blog/2022-08-29-delta-lake-drop-column/) so we may want to support the `on_schema_change = 'sync_all_columns'` config when columns are removed from the source (compared to the target).

![image](https://user-images.githubusercontent.com/8496748/213013430-b49a6b0b-9788-400e-997e-e57cb927ae8a.png)

![image](https://user-images.githubusercontent.com/8496748/213013730-2c52b2ae-eadc-45bc-b1e2-56f4afae0b6e.png)

### Describe alternatives you've considered

A user could probably achieve this today by rewriting some of the relevant built-in macros.

### Who will this benefit?

Previously a schema change (specifically a column removed from source vs target) will result in an exception:

https://github.com/dbt-labs/dbt-spark/blob/5ca20be56ec2d557b4fff5e42c320949040650d3/dbt/include/spark/macros/adapters.sql#L280

Primarily this will bring it up to par with other adapters's behaviour for `on_schema_change` without having users to implement their own overrides for the relevant helper macros (e.g. `alter_relation_add_remove_columns()` and family).

### Are you interested in contributing this feature?

Sure

### Anything else?

We did not support this behaviour back in https://github.com/dbt-labs/dbt-spark/pull/229 because delta could not drop columns - this is now supported (albeit with the necessary tblproperties applied).

Probably the way to do this is to retrieve the tblproperties of the target - and then decide whether to raise or not (or perhaps warn with a suggestion "in order to drop columns, please alter the tblproperties, ...".

We probably want this in dbt-databricks too. The behaviour in the adapter is the same as it is here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.