dbt-labs / dbt-labs/dbt

[v2 Bug] If there is a dbt command invoked (e.g. `dbt seed -t ci`) prior to `dbt build -s state:modified` - dbt v2 will not find the modified files

Open
#15,214 1 comment 0 reactions 1 assignee Claimed by @vgapeyev View on GitHub
adapter:snowflake area:engine discrepancy source:customer state status:triage type:bug
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this a new bug in dbt v2.x compared to the latest version of dbt 1.x?

- [x] I believe this is a new bug in dbt v2.x
- [x] I have searched the existing issues and could not find a duplicate

### Current Behavior

As per title - if you have a dbt job that runs in CI like so:

```sh
$ dbt seed -t ci # or any other dbt command (e.g. dbt ls)
$ dbt build -t ci -s state:modified --defer --state target_old
```

The fusion engine will report that there are no nodes selected / nothing to do - even when that is not true.

### Expected Behavior

The fusion engine should behave like the dbt-core engine and not incorrectly find "nothing to do".

### Steps To Reproduce

1. Install both fusion and core locally.

2. Setup 2 profiles:

```yaml
# /.dbt/profiles.yml
sf:
target: ci
outputs:
ci:
type: snowflake
schema: dbt_jyeo
...
prod:
type: snowflake
schema: prod
...
```

3. Setup the `dbt_project.yml`:

```yaml
# dbt_project.yml
name: dbt_basic_5
profile: sf
version: "1.0.0"

models:
dbt_basic_5:
+materialized: table
```

4. Then use this script to test both fusion and core runs:

> Make sure to change the path to your fusion/core install accordingly:

```sh
# repro.sh
# Usage:
# ./repro.sh # defaults to core
# ./repro.sh core # dbt 1.11 venv
# ./repro.sh fusion # ~/.local/bin/dbt
# Set your core / fusion bin accordingly below.

FLAVOR="${1:-core}"

case "$FLAVOR" in
core)
DBT_BIN="/Users/jeremy/git/dbt-basic/venv_dbt_1.11.latest/bin/dbt"
;;
fusion)
DBT_BIN="/Users/jeremy/.local/bin/dbt"
;;
*)
echo "Usage: $0 [core|fusion]"
exit 1
;;
esac

echo "========== Setup ${FLAVOR} =========="
alias dbt="$DBT_BIN"

[ -d "models" ] && rm -rf "models"
mkdir models
[ -d "seeds" ] && rm -rf "seeds"
mkdir seeds
echo "select 1 c" > models/foo.sql
echo "id\n1" > seeds/people.csv

echo "========== Building prod manifest =========="
[ -d "target" ] && rm -rf "target"
[ -d "target_old" ] && rm -rf "target_old"
dbt build -t prod
mv target target_old

echo "========== Add new model bar.sql =========="
echo "select 1 c" > models/bar.sql

echo "========== Change existing model foo.sql =========="
echo "select 2 c" > models/foo.sql
[ -d "target" ] && rm -rf "target" # delete target folder if accidentally created by lsp.

echo "========== Running CI steps =========="
dbt seed -t ci # This can be any command - e.g. `dbt compile -t ci`
echo "========== Running CI steps (foo and bar expected) =========="
dbt build -s state:modified -t ci --defer --state target_old
```

### Fusion

```sh
$ ./repro.sh fusion
========== Setup fusion ==========
========== Building prod manifest ==========
dbt-fusion 2.0.0-preview.178
Loading ~/.dbt/profiles.yml
Succeeded [ 2.19s] seed prod.people (table)
Succeeded [ 2.27s] model prod.foo (table)
New version available 2.0.0-preview.186 (run `dbt system update`)

====================================== Execution Summary ======================================
Finished 'build' successfully for target 'prod' [3.9s]
Processed: 1 model | 1 seed
Summary: 2 total | 2 success
========== Add new model bar.sql ==========
========== Change existing model foo.sql ==========
========== Running CI steps ==========
dbt-fusion 2.0.0-preview.178
Loading ~/.dbt/profiles.yml
Succeeded [ 1.64s] seed dbt_jyeo.people (table)
New version available 2.0.0-preview.186 (run `dbt system update`)

====================================== Execution Summary ======================================
Finished 'seed' successfully for target 'ci' [2.8s]
Processed: 1 seed
Summary: 1 total | 1 success
========== Running CI steps (foo and bar expected) ==========
dbt-fusion 2.0.0-preview.178
Loading ~/.dbt/profiles.yml

===================================== Errors and Warnings =====================================
[warning] [NoNodesForSelectionCriteria (dbt1092)]: The selection criterion 'state:modified' does not match any enabled nodes
[warning] [NoNodesSelected (dbt1601)]: Nothing to do. Try checking your model configs and model specification args

====================================== Execution Summary ======================================
Finished 'build' with 2 warnings for target 'ci' [289ms]
```

### Core

```sh
$ ./repro.sh core
========== Setup core ==========
========== Building prod manifest ==========
19:34:10 Running with dbt=1.11.10
19:34:10 Registered adapter: snowflake=1.11.5
19:34:10 Unable to do partial parsing because saved manifest not found. Starting full parse.
19:34:11 Found 1 model, 1 seed, 528 macros
19:34:11
19:34:11 Concurrency: 1 threads (target='prod')
19:34:11
19:34:12 1 of 2 START sql table model prod.foo .......................................... [RUN]
19:34:13 1 of 2 OK created sql table model prod.foo ..................................... [SUCCESS 1 in 1.02s]
19:34:13 2 of 2 START seed file prod.people ............................................. [RUN]
19:34:16 2 of 2 OK loaded seed file prod.people ......................................... [INSERT 1 in 2.59s]
19:34:16
19:34:16 Finished running 1 seed, 1 table model in 0 hours 0 minutes and 4.99 seconds (4.99s).
19:34:16
19:34:16 Completed successfully
19:34:16
19:34:16 Done. PASS=2 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=2
========== Add new model bar.sql ==========
========== Change existing model foo.sql ==========
========== Running CI steps ==========
19:34:18 Running with dbt=1.11.10
19:34:18 Registered adapter: snowflake=1.11.5
19:34:18 Unable to do partial parsing because saved manifest not found. Starting full parse.
19:34:19 Found 2 models, 1 seed, 528 macros
19:34:19
19:34:19 Concurrency: 2 threads (target='ci')
19:34:19
19:34:20 1 of 1 START seed file dbt_jyeo.people ......................................... [RUN]
19:34:23 1 of 1 OK loaded seed file dbt_jyeo.people ..................................... [INSERT 1 in 2.57s]
19:34:23
19:34:23 Finished running 1 seed in 0 hours 0 minutes and 3.75 seconds (3.75s).
19:34:23
19:34:23 Completed successfully
19:34:23
19:34:23 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=1
========== Running CI steps (foo and bar expected) ==========
19:34:24 Running with dbt=1.11.10
19:34:24 Registered adapter: snowflake=1.11.5
19:34:25 Found 2 models, 1 seed, 528 macros
19:34:25
19:34:25 Concurrency: 2 threads (target='ci')
19:34:25
19:34:26 1 of 2 START sql table model dbt_jyeo.bar ...................................... [RUN]
19:34:26 2 of 2 START sql table model dbt_jyeo.foo ...................................... [RUN]
19:34:27 1 of 2 OK created sql table model dbt_jyeo.bar ................................. [SUCCESS 1 in 1.05s]
19:34:27 2 of 2 OK created sql table model dbt_jyeo.foo ................................. [SUCCESS 1 in 1.22s]
19:34:27
19:34:27 Finished running 2 table models in 0 hours 0 minutes and 2.76 seconds (2.76s).
19:34:27
19:34:27 Completed successfully
19:34:27
19:34:27 Done. PASS=2 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=2
```

^ Observe how both `foo` and `bar` were rebuilt by core since `foo` and `bar` indeed were modified (foo changed and bar was new) compare to the deferred state. Fusion, however did not do that - the prior step during CI `dbt seed` caused it to misbehave - if you simply exclude that step then fusion will behave as expected...

```sh
# Comment out the `dbt seed -t ci` step in `repro.sh`
$ ./repro.sh fusion
========== Setup fusion ==========
========== Building prod manifest ==========
dbt-fusion 2.0.0-preview.178
Loading ~/.dbt/profiles.yml
Succeeded [ 1.63s] model prod.foo (table)
Succeeded [ 1.88s] seed prod.people (table)
New version available 2.0.0-preview.186 (run `dbt system update`)

============================================== Execution Summary ===============================================
Finished 'build' successfully for target 'prod' [3.1s]
Processed: 1 model | 1 seed
Summary: 2 total | 2 success
========== Add new model bar.sql ==========
========== Change existing model foo.sql ==========
========== Running CI steps ==========
========== Running CI steps (foo and bar expected) ==========
dbt-fusion 2.0.0-preview.178
Loading ~/.dbt/profiles.yml
Succeeded [ 1.39s] model dbt_jyeo.foo (table)
Succeeded [ 1.59s] model dbt_jyeo.bar (table)
New version available 2.0.0-preview.186 (run `dbt system update`)

============================================== Execution Summary ===============================================
Finished 'build' successfully for target 'ci' [2.9s]
Processed: 2 models
Summary: 2 total | 2 success
```

### Relevant log output

```shell

```

### Environment

```markdown
- OS: macOS
- CPU: ARM
- dbt distribution and version:

Core:
- installed: 1.11.10
- latest: 1.11.11 - Update available!

Your version of dbt-core is out of date!
You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation

Plugins:
- snowflake: 1.11.5 - Up to date!

----

dbt-fusion 2.0.0-preview.178 (similar behaviour on dbt-fusion 2.0.0-preview.186)
```

### Which database adapter are you using?

snowflake

### Is this a discrepancy vs. dbt 1.x?

- [x] Yes — this works in dbt 1.x but not in dbt v2.x

### Additional Context

P.s. It's a pretty common operation for users to have sequential steps in their CI jobs like demonstrated:

Step 1: In CI jobs - seed all seeds into the PR schema.
Step 2: Build modified models.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.