dbt-labs / dbt-labs/dbt

Should column level `meta` modifications be considered `state:modified`?

Open
#10,189 11 comments 0 reactions 0 assignees View on GitHub
Refinement type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

As title, changes to column level `meta` configs aren't considered `state:modified`.

### Expected Behavior

Changes to column level `meta` configs should be considered `state:modified`.

### Steps To Reproduce

1. Setup dbt project.

```yml
# dbt_project.yml
name: my_dbt_project
profile: all
config-version: 2
version: "1.0.0"

models:
my_dbt_project:
+materialized: table
```

```yml
# models/schema.yml
version: 2
models:
- name: foo
config:
meta:
k: v
columns:
- name: id
meta:
k: v
- name: bar
config:
meta:
k: v
columns:
- name: id
meta:
k: v
```

```sql
-- models/foo.sql
select 1 id

-- models/bar.sql
select 1 id
```

2. Initial build and state storing.

```sh
$ dbt build && mv target target_old
00:00:49 Running with dbt=1.7.14
00:00:49 Registered adapter: postgres=1.7.14
00:00:49 Found 2 models, 0 sources, 0 exposures, 0 metrics, 402 macros, 0 groups, 0 semantic models
00:00:49
00:00:49 Concurrency: 4 threads (target='dev')
00:00:49
00:00:49 1 of 2 START sql table model public.bar ........................................ [RUN]
00:00:49 2 of 2 START sql table model public.foo ........................................ [RUN]
00:00:49 1 of 2 OK created sql table model public.bar ................................... [SELECT 1 in 0.08s]
00:00:49 2 of 2 OK created sql table model public.foo ................................... [SELECT 1 in 0.08s]
00:00:49
00:00:49 Finished running 2 table models in 0 hours 0 minutes and 0.17 seconds (0.17s).
00:00:49
00:00:49 Completed successfully
00:00:49
00:00:49 Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2
```

3. Modify `schema.yml` file (a model level change on `foo`, and a column level change on `bar`):

```yml
# models/schema.yml
version: 2
models:
- name: foo
config:
meta:
k: model_level_change
columns:
- name: id
meta:
k: v
- name: bar
config:
meta:
k: v
columns:
- name: id
meta:
k: column_level_change
```

4. Deferred build:

```sh
$ dbt build -s state:modified --defer --state target_old
00:02:48 Running with dbt=1.7.14
00:02:48 Registered adapter: postgres=1.7.14
00:02:48 Unable to do partial parsing because saved manifest not found. Starting full parse.
00:02:48 Found 2 models, 0 sources, 0 exposures, 0 metrics, 402 macros, 0 groups, 0 semantic models
00:02:48
00:02:48 Concurrency: 4 threads (target='dev')
00:02:48
00:02:48 1 of 1 START sql table model public.foo ........................................ [RUN]
00:02:48 1 of 1 OK created sql table model public.foo ................................... [SELECT 1 in 0.06s]
00:02:48
00:02:48 Finished running 1 table model in 0 hours 0 minutes and 0.15 seconds (0.15s).
00:02:48
00:02:48 Completed successfully
00:02:48
00:02:48 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
```

> `foo` is built but not `bar`.

### Relevant log output

```shell
Shared above.
```

### Environment

```markdown
- OS: macOS
- Python: 3.11.9
- dbt:

Core:
- installed: 1.7.14
- latest: 1.8.0 - Update available!

Plugins:
- postgres: 1.7.14 - Update available!
```

### Which database adapter are you using with dbt?

postgres

### Additional Context

Not 100% sure if this has ever worked - could be FR like https://github.com/dbt-labs/dbt-core/issues/10020

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.