dbt-labs / dbt-labs/dbt

[Bug] If a test / model config is added (to a `schema.yml`) before a model is written (`model.sql`), partial parsing will not find the test / model config until a full reparse

Open
#10,461 5 comments 0 reactions 0 assignees View on GitHub
dbt tests engine:v1 partial_parsing type:bug
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

If you wrote a test first, parse the project, then wrote the model, dbt will not actually run the test due to partial parsing.

### Expected Behavior

Even with partial parsing, dbt should know to run the test even if written prior to the model file.

### Steps To Reproduce

1. Add a schema yml file first but importantly don't create any model file yet.

```yaml
# models/schema.yml
models:
- name: foo
columns:
- name: id
data_tests:
- not_null
```

2. Do initial parse with clean `target/`:

```sh
$ ls target
ls: target: No such file or directory

$ dbt parse
20:30:13 Running with dbt=1.8.3
20:30:13 Registered adapter: postgres=1.8.2
20:30:13 Unable to do partial parsing because saved manifest not found. Starting full parse.
20:30:14 [WARNING]: Did not find matching node for patch with name 'foo' in the 'models' section of file 'models/schema.yml'
20:30:14 [WARNING]: Test 'test.my_dbt_project.not_null_foo_id.f099b1e59c' (models/schema.yml) depends on a node named 'foo' in package '' which was not found
20:30:14 [WARNING]: Configuration paths exist in your dbt_project.yml file which do not apply to any resources.
There are 1 unused configuration paths:
- models.my_dbt_project
20:30:14 Performance info: /Users/jeremy/git/dbt-basic/target/perf_info.json
```

3. Add model file:

```sql
-- models/foo.sql
select 1 id
```

4. Build:

```sh
$ dbt build
20:31:35 Running with dbt=1.8.3
20:31:35 Registered adapter: postgres=1.8.2
20:31:35 Found 1 model, 413 macros
20:31:35
20:31:35 Concurrency: 4 threads (target='pg')
20:31:35
20:31:35 1 of 1 START sql table model public.foo ........................................ [RUN]
20:31:35 1 of 1 OK created sql table model public.foo ................................... [SELECT 1 in 0.07s]
20:31:35
20:31:35 Finished running 1 table model in 0 hours 0 minutes and 0.19 seconds (0.19s).
20:31:35
20:31:35 Completed successfully
20:31:35
20:31:35 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
```

^ Notice no "test" was identified.

5. Build but do a full parse:

```sh
$ dbt build --no-partial-parse
20:32:22 Running with dbt=1.8.3
20:32:22 Registered adapter: postgres=1.8.2
20:32:23 Found 1 model, 1 test, 413 macros
20:32:23
20:32:23 Concurrency: 4 threads (target='pg')
20:32:23
20:32:23 1 of 2 START sql table model public.foo ........................................ [RUN]
20:32:23 1 of 2 OK created sql table model public.foo ................................... [SELECT 1 in 0.06s]
20:32:23 2 of 2 START test not_null_foo_id .............................................. [RUN]
20:32:23 2 of 2 PASS not_null_foo_id .................................................... [PASS in 0.04s]
20:32:23
20:32:23 Finished running 1 table model, 1 test in 0 hours 0 minutes and 0.24 seconds (0.24s).
20:32:23
20:32:23 Completed successfully
20:32:23
20:32:23 Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2
```

### Relevant log output

_No response_

### Environment

```markdown
- OS: macOS
- Python: 3.11.9
- dbt: 1.8.3
```

### Which database adapter are you using with dbt?

postgres

### Additional Context

In dbt Cloud, every save action does a parse. So if a user was practicing TDD - I guess write the test first :P then they could run into this scenario.

https://dbt-labs.slack.com/archives/C02SRNY2EQ4/p1721245567642059

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.