dbt-labs / dbt-labs/dbt-adapters
[CT-1842] [Feature] Support `on_schema_change='sync_all_columns'` for Delta tables
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this your first time submitting a feature request?
- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt-spark functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
Delta Lake 2.0 supports dropping of columns on delta tables if the table has certain `tblproperties` (https://delta.io/blog/2022-08-29-delta-lake-drop-column/) so we may want to support the `on_schema_change = 'sync_all_columns'` config when columns are removed from the source (compared to the target).


### Describe alternatives you've considered
A user could probably achieve this today by rewriting some of the relevant built-in macros.
### Who will this benefit?
Previously a schema change (specifically a column removed from source vs target) will result in an exception:
https://github.com/dbt-labs/dbt-spark/blob/5ca20be56ec2d557b4fff5e42c320949040650d3/dbt/include/spark/macros/adapters.sql#L280
Primarily this will bring it up to par with other adapters's behaviour for `on_schema_change` without having users to implement their own overrides for the relevant helper macros (e.g. `alter_relation_add_remove_columns()` and family).
### Are you interested in contributing this feature?
Sure
### Anything else?
We did not support this behaviour back in https://github.com/dbt-labs/dbt-spark/pull/229 because delta could not drop columns - this is now supported (albeit with the necessary tblproperties applied).
Probably the way to do this is to retrieve the tblproperties of the target - and then decide whether to raise or not (or perhaps warn with a suggestion "in order to drop columns, please alter the tblproperties, ...".
We probably want this in dbt-databricks too. The behaviour in the adapter is the same as it is here.
Contributor guide
Assessment
This issue has not been assessed yet.