[v2 Bug] Fusion: unit test on an incremental model fails with "Missing cached schema" (dbt9002); same test passes if the model is a table/view, and passes under dbt-core
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Is this a new bug in dbt v2.x compared to the latest version of dbt 1.x?
- [x] I believe this is a new bug in dbt v2.x
- [x] I have searched the existing issues and could not find a duplicate
### Current Behavior
unit test on a model with `materialized = 'incremental'` fails:
```
[error] [Unexpected (dbt9002)]: Missing cached schema for unit test 'unit_test.my_project.my_incremental_model.my_incremental_model_test'
```
Change nothing but `materialized` — to `table` or `view` — and the same test on the same SQL passes.
Run the incremental case under dbt-core (1.12.0-b2) and it passes there too.
Two further details about the cache the error names:
- **It is `target/metadata/warehouse/schemas/*.parquet`**, keyed by a `lookup_key` column of the form
`Frontier(..)` (other columns: `sdf_schema`, `original_schema`,
`cached_at_ms`, `build_hash`, `epoch`). After a failing run it contains entries for the test's
`given` input relations but none for the model under test.
- **No command repopulates it.** `dbt compile -s ` succeeds and leaves the file
byte-identical; what it writes instead is a *different namespace under a different key form*,
`target/metadata/compile/schemas/` keyed `Selected(model..)`. `dbt run`,
re-running `dbt test`, and recreating the relation with `create or replace table` also do not help.
Separately, and probably related: **deleting `target/metadata/` makes every unit test in the project
fail with this same error**, and nothing restores it — so that directory does not behave like a
regenerable cache.
The error is also raised before fixtures are rendered, so it pre-empts and hides whatever the unit
test would actually have reported.
### Expected Behavior
A unit test on an incremental model should run, as it does under dbt-core and as it does for the same
model materialized as a table or view.
On a cache miss Fusion should fetch the relation's schema from the warehouse — the relation exists and
is queryable — rather than failing. If it genuinely cannot, the error should be actionable rather than
`Unexpected`: name the cache key it could not resolve and the command that would populate it. At
present there is no documented way to recover.
### Steps To Reproduce
Two models with identical SQL, differing only in `materialized`, plus one unit-test YAML.
`models/repro/my_incremental_model.sql`:
```sql
{{ config(materialized = 'incremental') }}
select 'a1' as id, 'green' as color
union all
select 'a2' as id, 'blue' as color
```
`models/repro/my_table_model.sql`:
```sql
{{ config(materialized = 'table') }}
select 'a1' as id, 'green' as color
union all
select 'a2' as id, 'blue' as color
```
`models/repro/_repro__unit_tests.yml`:
```yaml
unit_tests:
- name: my_incremental_model_test
model: my_incremental_model
overrides:
macros:
is_incremental: false
given: []
expect:
rows:
- {id: a1, color: green}
- {id: a2, color: blue}
- name: my_table_model_test
model: my_table_model
given: []
expect:
rows:
- {id: a1, color: green}
- {id: a2, color: blue}
```
Then:
```bash
dbt run --target dev -s my_incremental_model my_table_model # both build fine
dbt test --target dev --select "my_incremental_model" # dbt9002
dbt test --target dev --select "my_table_model" # passes
```
Both relations are built before testing, so each matches its own config — this is not a
config/relation mismatch. `materialized = 'view'` also passes. Adding a `given` input to the
incremental model's test does not help. The same incremental case passes under dbt-core.
### Relevant log output
```shell
Full transcripts, with the invocations. Identifiers are anonymised; `on-run-start`/`on-run-end` hook
lines and the BigQuery usage banner are elided where marked, and nothing else is trimmed.
**Building both models first — no problem here:**
$ dbt run --target dev -s my_incremental_model my_table_model
dbt-fusion 2.0.0-preview.209
Loading ~/.dbt/profiles.yml
Loading packages.yml
[... on-run-start hooks ...]
[... BigQuery Usage Summary ...]
[... on-run-end hooks ...]
==================== Execution Summary =====================
Finished 'run' successfully for target 'dev' [1m 15s]
Summary: 5 total | 5 success
**The incremental model's unit test — fails:**
$ dbt clean && dbt deps
$ dbt test --target dev --select "test_type:unit,my_incremental_model"
dbt-fusion 2.0.0-preview.209
Loading ~/.dbt/profiles.yml
Loading packages.yml
[... on-run-start hooks ...]
Failed [ 0.00s] unit_test my_dataset_dbt_test__audit.my_incremental_model_test (models/repro/_repro__unit_tests.yml:2:11)
[... BigQuery Usage Summary: 4 jobs, 10.00 MB billed ...]
[... on-run-end hooks ...]
=================== Errors and Warnings ====================
[error] [Unexpected (dbt9002)]: Missing cached schema for unit test 'unit_test.my_project.my_incremental_model.my_incremental_model_test'
==================== Execution Summary =====================
Finished 'test' with 1 error for target 'dev' [1m 22s]
Processed: 3 hooks | 1 unit test
Summary: 4 total | 3 success | 1 error
**The table model's unit test, identical SQL — passes:**
$ dbt clean && dbt deps
$ dbt test --target dev --select "test_type:unit,my_table_model"
dbt-fusion 2.0.0-preview.209
[... as above ...]
Passed [ 5.54s] unit_test my_dataset_dbt_test__audit.my_table_model_test
==================== Execution Summary =====================
Finished 'test' successfully for target 'dev'
Summary: 4 total | 4 success
Note the timing difference: the failing case returns in `0.00s`, i.e. before any warehouse round-trip,
while the passing case takes ~5s. On the very first run after a `dbt clean` the failure takes
0.2–1.0s; on subsequent runs it is instantaneous.
**`dbt compile` on the affected model — succeeds, and does not fix it:**
$ dbt compile --target dev -s my_incremental_model
==================== Execution Summary =====================
Finished 'compile' successfully for target 'dev' [1m 11s]
Processed: 3 hooks | 1 model | 5 tests
Summary: 9 total | 9 success
$ md5 -q target/metadata/warehouse/schemas/0.parquet # before and after: identical
8a29a29b31fdcfbbf1a7e5b14f1841b5
**The same unit test under dbt-core — passes.** (Invoked as `dbtc` on this machine because Fusion owns
the `dbt` entrypoint; it is a stock dbt-core install.)
$ dbtc test --target dev --select "test_type:unit,my_incremental_model"
23:31:58 Running with dbt=1.12.0-b2
23:31:59 Registered adapter: bigquery=1.11.3
02:38:09 1 of 1 PASS my_incremental_model::my_incremental_model_test [PASS in 10.94s]
02:38:16 Completed successfully
02:38:16 Done. PASS=1 WARN=0 ERROR=0 SKIP=0 NO-OP=0 REUSED=0 TOTAL=1
**Reading the cache** (it is compressed parquet, so `strings` returns nothing — that is a dead end,
not an empty file):
$ python3 -c "
import pyarrow.parquet as pq
t = pq.read_table('target/metadata/warehouse/schemas/0.parquet')
print([f'{f.name}: {f.type}' for f in t.schema])
print(sorted(t.to_pydict()['lookup_key']))
"
['lookup_key: string', 'sdf_schema: binary', 'original_schema: binary',
'cached_at_ms: int64', 'build_hash: string', 'epoch: int32']
['Frontier(my_project.my_dataset.some_given_input_relation)', ...]
```
### Environment
```markdown
- **OS:** macOS 14.7.4 (build 23H420), Darwin 23.6.0, arm64 (Apple Silicon)
- **Fusion:** `2.0.0-preview.209`. **Not version-dependent** — `preview.205` and `preview.210`
reproduce identically. Consistent with `CHANGELOG-fusion.md` having no `dbt9002` or cached-schema
entry through `preview.212`.
- **Compared against:** `dbt-core 1.12.0-b2` with `dbt-bigquery 1.11.3`
- Local dev target; our CI and production runs are unaffected (production does not run unit tests).
```
### Which database adapter are you using?
bigquery
### Is this a discrepancy vs. dbt 1.x?
- [x] Yes — this works in dbt 1.x but not in dbt v2.x
### Additional Context
**`materialized = 'incremental'` may be necessary but not sufficient**, so it is possible this does not
reproduce for you first try. In the project where we hit it, 4 of 5 unit-tested incremental models fail
this way but one does not — it uses `incremental_strategy = 'merge'` with `unique_key` and
`incremental_predicates`, and its unit tests pass from a cold cache. Mutating a failing model toward
that shape did not rescue it (we tried removing a static `partitions` config, switching
`insert_overwrite` → `merge`, and adding `unique_key`; only changing `materialized` fixed it). So there
may be a narrower condition we did not isolate.
**One behaviour that makes this look intermittent in a real project:** a relation whose schema Fusion
will not resolve as the *model under test* is resolved successfully when that same relation appears as
a `given` **input** of some other unit test. From a cold cache, running a test whose input is relation
R fails, but caches R — after which R's own unit test passes, having failed moments before. So in a
project with many unit tests, whether a given test fails depends on selection order, and the number of
failures varies between runs. Models that are not any other test's input fail every time.
The closest existing changelog entry is `preview.202`, *"Fix stale `sourced_remote` schema cache for
DuckDB adapter causing unit test fixture column mismatches after model schema changes"* — different
adapter and a different symptom, but it suggests this cache has had adapter-specific problems before.
Contributor guide
Assessment
This issue has not been assessed yet.