dbt-labs / dbt-labs/dbt-project-evaluator

Rules aggregate by resource_name across packages, causing false positives

Open
#572 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
577
Forks
93
Avg merge
9h 46m
Merged PRs (30d)
1

Description

## Describe the bug

When a dbt project contains multiple packages that define resources with the same `resource_name`, some `dbt_project_evaluator` rules appear to aggregate across packages and produce false positives.

From reading the package code, it looks like several DAG-based rules operate on `resource_name` rather than a package-scoped identifier. As a result, resources from different packages that happen to share the same name can be treated as the same node.

## Examples of affected behavior

I confirmed this with examples shaped like the following:

- Package A has a staging model `stg_shared_name` that depends on an internal base model.
- Package B has another staging model with the same name `stg_shared_name`, but this one depends directly on a source.

In this setup, `Direct Join to Source` appears to report `stg_shared_name` even though the Package A model does not directly depend on a source.

I also confirmed a similar pattern for `Source Fanout`:

- Package A has a source `shared_source.table_x` and a model `stg_metrics`
- Package B has a source with the same name `shared_source.table_x` and a model with the same name `stg_metrics`

Each package individually has a 1:1 source-to-model relationship, but `Source Fanout` appears to aggregate them together and report a false positive.

I suspect `Models with Too Many Joins` and possibly other rules that aggregate by `resource_name` may be affected as well, although I have only directly verified the two cases above.

## Expected behavior

Resources defined in different packages should be treated as distinct graph nodes, even if they share the same `resource_name`.

## Actual behavior

Some rules appear to aggregate across packages based on `resource_name`, which causes false positives when identical names exist in multiple packages.

## Steps to reproduce

A minimal reproduction would look like this:

1. Create two dbt packages in the same project.
2. In both packages, define resources with the same model name and/or source name.
3. Make their lineage different across packages.
4. Run `dbt_project_evaluator`.

## Why this seems to happen

From reading the implementation, it looks like graph/resource models derive and propagate `resource_name`, and some downstream rules group or join on that value. If package identity is not included in the effective key, same-named resources across packages can collide.

## Impact

This makes it difficult to rely on evaluator output in monorepos or multi-package dbt projects where naming conventions are reused across packages.

## Workaround

As a local workaround, we filtered specific rules after `evaluation_results`, but that only helps for some false positives and does not solve cases where aggregated metrics (for example join counts) are already incorrect upstream.

## Environment

- dbt version: 1.11.7
- adapter: BigQuery
- dbt_project_evaluator version: current release in use

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.