dbt-labs / dbt-labs/dbt

get_manifest_artifacts: Compilation Error in downstream project when upstream project adds versioning to public model referenced in intermediate project

Open
#11,032 4 comments 3 reactions 0 assignees View on GitHub
engine:v1 model_versions multi_project type:bug
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

When adding a versioned model to Project A, whereas Project B references that model via cross-project ref, and Project C includes both Project A & Project B dependencies.yml we can confirm that Project C will error once Project A has deployed the version model change to production, but Project B has not yet reran the cross-project ref within Project B's production environment. This causes production runs to fail downstream, without having those dependencies included in the jobs, due to the publication artifact not being updated with the latest node.

We try to inject dependencies among public models in upstream projects, so that downstream projects can see if one of their upstream parents depends on another of their upstream parents
so a project that depends on two other projects (e.g. Project A + Project B) could see this error, even if it doesn't depend on either of those models directly -
and this error might happen if a public model that was an upstream dependency of another public model, has its access changed (no longer public) or is disabled/deleted/missing.

### Expected Behavior

Options:
- just fail silently, rather than raising an error
I think the risk of this approach is that we fail to find real node-level cycles across projects.
or
- should we try to transform the public_model_dependency of proj_B.model_Y to match the latest version of proj_A.model_X, now that we know it's versioned?

### Steps To Reproduce

1. Setup dependencies.yml in Project B as:
```yaml
# dependencies.yml
projects:
- name: project_a
```
- create an xref model `select * from {{ ref(project_a, model_x) }}`
- ensure access as public for xref model
2. Setup dependencies.yml in Project C as:
```yaml
# dependencies.yml
projects:
- name: project_a
- name: project_b
```
3. Ensure project_a model being xref'd access is public and run production build job for successful build.
4. Run project_b production build job including xref model.
5. in project_a, change xref'd model_x to specify version

```yaml
# schema.yml example

version: 2

models:
- name: my_first_dbt_model
access: public
latest_version: 1
description: "A starter dbt model"
columns:
- name: id
description: "The primary key for this table"
tests:
- unique
- not_null
versions:
- v: 1
```

7. Commit, merge, and run project_a production build job to ensure versioned model is picked up.
8. Try running project_c production build job now and it will fail.

note: Running project_b production build job will update the manifest and grab the latest version/updated node, whereas then project_c production build will successfully complete without error.

### Relevant log output

```shell
2024-11-20 21:56:03.149660 (MainThread): 21:56:03 Encountered an error:
Runtime Error
get_manifest_artifacts: Compilation Error
'model.mesh_test.meshy' depends on 'model.my_new_project.my_first_dbt_model' which is not in the graph!
2024-11-20 21:56:03.150532 (MainThread): 21:56:03 Resource report: {"command_name": "build", "command_wall_clock_time": 2.1111817, "process_user_time": 3.218299, "process_kernel_time": 0.367934, "process_mem_max_rss": "211344", "process_out_blocks": "1976", "command_success": false, "process_in_blocks": "0"}
2024-11-20 21:56:03.151034 (MainThread): 21:56:03 Observability Metric: command_success=0.0
2024-11-20 21:56:03.151577 (MainThread): 21:56:03 Observability Metric: command_wall_clock_time=2.1111817359924316
2024-11-20 21:56:03.152051 (MainThread): 21:56:03 Observability Metric: process_user_time=3.21829891204834
2024-11-20 21:56:03.152459 (MainThread): 21:56:03 Observability Metric: process_kernel_time=0.367933988571167
2024-11-20 21:56:03.152850 (MainThread): 21:56:03 Observability Metric: process_mem_max_rss=211344.0
2024-11-20 21:56:03.153441 (MainThread): 21:56:03 Command `dbt build` failed at 21:56:03.153347 after 2.11 seconds
```

### Environment

```markdown
- dbt: versionless
```

### Which database adapter are you using with dbt?

_No response_

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.