dbt-labs / dbt-labs/dbt-adapters
[Feature] Apply constraints for incrementals even if they are not being created for the first time / is a subsequent run of the model
- Dominant language
- Python
- Stars
- 233
- Forks
- 362
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 9
Description
### Is this your first time submitting a feature request?
- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
At the moment, it seems that constraints only apply during ctas and not otherwise (at least for postgres and databricks adapters). This means that the following sequence will not add the constraint:
1. Add incremental model to project but don't specify constraint yet.
```sql
-- models/foo.sql
{{ config(materialized='incremental', on_schema_change='fail') }}
select 1 id
```
2. Build the model so that `foo` exist on the dwh.
3. Add constraint to the model:
```yaml
# models/schema.yml
models:
- name: foo
config:
contract:
enforced: true
columns:
- name: id
data_type: int
constraints:
- type: not_null
```
4. Build foo again:
```sh
$ dbt build
23:01:22 1 of 1 START sql incremental model public.foo .................................. [RUN]
23:01:22 Re-using an available connection from the pool (formerly list_postgres_public, now model.my_dbt_project.foo)
23:01:22 Began compiling node model.my_dbt_project.foo
23:01:22 Writing injected SQL for node "model.my_dbt_project.foo"
23:01:22 Began executing node model.my_dbt_project.foo
23:01:22 Using postgres connection "model.my_dbt_project.foo"
23:01:22 On model.my_dbt_project.foo: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.foo"} */
select * from (
select 1 id
) as __dbt_sbq
where false
limit 0
23:01:22 Opening a new connection, currently in state closed
23:01:22 SQL status: SELECT 0 in 0.009 seconds
23:01:22 Using postgres connection "model.my_dbt_project.foo"
23:01:22 On model.my_dbt_project.foo: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.foo"} */
select * from (
select
cast(null as int)
as id
) as __dbt_sbq
where false
limit 0
23:01:22 SQL status: SELECT 0 in 0.002 seconds
23:01:22 Using postgres connection "model.my_dbt_project.foo"
23:01:22 On model.my_dbt_project.foo: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.foo"} */
create temporary table "foo__dbt_tmp110122647893"
as
(
select 1 id
);
23:01:22 SQL status: SELECT 1 in 0.005 seconds
23:01:22 Using postgres connection "model.my_dbt_project.foo"
23:01:22 On model.my_dbt_project.foo: BEGIN
23:01:22 SQL status: BEGIN in 0.001 seconds
23:01:22 Using postgres connection "model.my_dbt_project.foo"
23:01:22 On model.my_dbt_project.foo: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.foo"} */
select
column_name,
data_type,
character_maximum_length,
numeric_precision,
numeric_scale
from INFORMATION_SCHEMA.columns
where table_name = 'foo__dbt_tmp110122647893'
order by ordinal_position
23:01:22 SQL status: SELECT 1 in 0.019 seconds
23:01:22 Using postgres connection "model.my_dbt_project.foo"
23:01:22 On model.my_dbt_project.foo: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.foo"} */
select
column_name,
data_type,
character_maximum_length,
numeric_precision,
numeric_scale
from "postgres".INFORMATION_SCHEMA.columns
where table_name = 'foo'
and table_schema = 'public'
order by ordinal_position
23:01:22 SQL status: SELECT 1 in 0.003 seconds
23:01:22
In "postgres"."public"."foo":
Schema changed: False
Source columns not in target: []
Target columns not in source: []
New column types: []
23:01:22 Writing runtime sql for node "model.my_dbt_project.foo"
23:01:22 Using postgres connection "model.my_dbt_project.foo"
23:01:22 On model.my_dbt_project.foo: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.foo"} */
insert into "postgres"."public"."foo" ("id")
(
select "id"
from "foo__dbt_tmp110122647893"
)
23:01:22 SQL status: INSERT 0 1 in 0.001 seconds
23:01:22 On model.my_dbt_project.foo: COMMIT
23:01:22 Using postgres connection "model.my_dbt_project.foo"
23:01:22 On model.my_dbt_project.foo: COMMIT
23:01:22 SQL status: COMMIT in 0.001 seconds
23:01:22 On model.my_dbt_project.foo: Close
23:01:22 Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '19490200-b1d7-4bef-8530-4632db0be649', 'context': []}
23:01:22 1 of 1 OK created sql incremental model public.foo ............................. [INSERT 0 1 in 0.14s]
```
> `foo` does not get the not null id constraint added to it.
To get the constraint added, it's necessary to do a ctas:
```sh
$ dbt build --full-refresh
04:03:41 1 of 1 START sql incremental model public.foo .................................. [RUN]
04:03:41 Re-using an available connection from the pool (formerly list_postgres_public, now model.my_dbt_project.foo)
04:03:41 Began compiling node model.my_dbt_project.foo
04:03:41 Writing injected SQL for node "model.my_dbt_project.foo"
04:03:41 Began executing node model.my_dbt_project.foo
04:03:41 Using postgres connection "model.my_dbt_project.foo"
04:03:41 On model.my_dbt_project.foo: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.foo"} */
select * from (
select 1 id
) as __dbt_sbq
where false
limit 0
04:03:41 Opening a new connection, currently in state closed
04:03:41 SQL status: SELECT 0 in 0.009 seconds
04:03:41 Using postgres connection "model.my_dbt_project.foo"
04:03:41 On model.my_dbt_project.foo: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.foo"} */
select * from (
select
cast(null as int)
as id
) as __dbt_sbq
where false
limit 0
04:03:41 SQL status: SELECT 0 in 0.001 seconds
04:03:41 Writing runtime sql for node "model.my_dbt_project.foo"
04:03:41 Using postgres connection "model.my_dbt_project.foo"
04:03:41 On model.my_dbt_project.foo: BEGIN
04:03:41 SQL status: BEGIN in 0.001 seconds
04:03:41 Using postgres connection "model.my_dbt_project.foo"
04:03:41 On model.my_dbt_project.foo: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.foo"} */
create table "postgres"."public"."foo__dbt_tmp"
(
id int not null
)
;
insert into "postgres"."public"."foo__dbt_tmp" (
id
)
(
select id
from (
select 1 id
) as model_subq
);
04:03:41 SQL status: INSERT 0 1 in 0.002 seconds
04:03:41 Using postgres connection "model.my_dbt_project.foo"
04:03:41 On model.my_dbt_project.foo: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.foo"} */
alter table "postgres"."public"."foo" rename to "foo__dbt_backup"
04:03:41 SQL status: ALTER TABLE in 0.001 seconds
04:03:41 Using postgres connection "model.my_dbt_project.foo"
04:03:41 On model.my_dbt_project.foo: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.foo"} */
alter table "postgres"."public"."foo__dbt_tmp" rename to "foo"
04:03:41 SQL status: ALTER TABLE in 0.001 seconds
04:03:41 On model.my_dbt_project.foo: COMMIT
04:03:41 Using postgres connection "model.my_dbt_project.foo"
04:03:41 On model.my_dbt_project.foo: COMMIT
04:03:41 SQL status: COMMIT in 0.002 seconds
04:03:41 Applying DROP to: "postgres"."public"."foo__dbt_backup"
04:03:41 Using postgres connection "model.my_dbt_project.foo"
04:03:41 On model.my_dbt_project.foo: /* {"app": "dbt", "dbt_version": "1.10.0b1", "profile_name": "all", "target_name": "pg", "node_id": "model.my_dbt_project.foo"} */
drop table if exists "postgres"."public"."foo__dbt_backup" cascade
04:03:41 SQL status: DROP TABLE in 0.003 seconds
04:03:41 On model.my_dbt_project.foo: Close
04:03:41 Sending event: {'category': 'dbt', 'action': 'run_model', 'label': '8b583181-7053-4cda-a349-e4014222e07e', 'context': []}
04:03:41 1 of 1 OK created sql incremental model public.foo ............................. [INSERT 0 1 in 0.09s]
```
It can be costly for large tables to full refresh this just to add constraints. Ideally, we would detect that the table does not have the not null id constraint that the yaml has specified and then apply it.
### Describe alternatives you've considered
Manually update via running an out of band/dbt sql statement to bring the "state of the world" in sync:
```sql
alter table foo alter column id set not null;
```
### Who will this benefit?
Folks who may be confused as to why they have added constraints to their yamls and see that it does not do anything to the actual relation in the dwh.
### Are you interested in contributing this feature?
Yes
### Anything else?
Tested this on postgres and databricks. I think this should be pretty straightforward to apply by modifying the incremental materializations to - if contract enforced is true, then always emit the DDL to add constraints (perhaps check info schema to see if emitting DDL is necessary).
Contributor guide
Assessment
This issue has not been assessed yet.