dbt-labs / dbt-labs/dbt-codegen
Bug: Any source table that overrides `identifier:` raises a compilation error and no template is produced.
- Dominant language
- Makefile
- Stars
- 676
- Forks
- 132
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
`generate_unit_test_template` generates `source()` references using the source node's **`identifier`**, which is the physical relation name, instead of its **`name`**, which is the logical source name declared in YAML.
Because dbt resolves `source()` using the declared `name`, the macro fails whenever a source overrides `identifier:`. The operation raises a compilation error and does not generate a unit test template.
This affects the primary use case for `identifier:`: allowing the physical warehouse table name to differ from the name used by dbt.
The issue is easy to miss because `identifier` defaults to `name`. For sources that do not explicitly override `identifier`, both values are the same and the macro works as expected.
### Steps to reproduce
**1. Declare a source whose logical name differs from its physical table name:**
```yaml
# models/staging/epicor/sources/_epicor_part.yml
version: 2
sources:
- name: epicor_erp
database: data_warehouse_sources
schema: epicor_prod
tables:
- name: part # Logical name used by source()
identifier: erp__part # Physical relation in the warehouse
```
**2. Create a model that selects from the source:**
```sql
-- models/staging/epicor/base_epicor__part.sql
with source as (
select * from {{ source('epicor_erp', 'part') }}
),
renamed as (
select
nullif(trim(lower(company)), '') as company,
nullif(trim(lower(part_num)), '') as part_number
from source
)
select * from renamed
```
**3. Run the generator:**
```shell
dbt run-operation generate_unit_test_template \
--args '{"model_name": "base_epicor__part", "inline_columns": true}'
```
### Expected results
The generated unit test should reference the source using its declared logical name:
```yaml
unit_tests:
- name: unit_test_base_epicor__part
model: base_epicor__part
given:
- input: source("epicor_erp", "part")
rows:
- {company: , part_num: }
expect:
rows:
- {company: , part_number: }
```
### Actual results
The operation fails with a compilation error, and no template is generated.
The macro attempts to resolve:
```jinja
source("epicor_erp", "erp__part")
```
However, the source is declared in dbt as:
```jinja
source("epicor_erp", "part")
```
In other words, the macro uses the physical `identifier` instead of the logical source `name`.
### Screenshots and log output
```text
22:13:34 Running with dbt=1.12.0
22:13:34 Registered adapter: trino=1.10.3
22:13:35 Found 1 operation, 49 models, 1 seed, 22 data tests, 2 sources, 1172 macros
22:13:35 [ERROR]: Encountered an error while running operation: Compilation Error
Macro 'macro.codegen.generate_unit_test_template' (macros/generate_unit_test_template.sql)
depends on a source named 'epicor_erp.erp__part' which was not found
> in macro default__generate_unit_test_template (macros/generate_unit_test_template.sql)
> called by macro generate_unit_test_template (macros/generate_unit_test_template.sql)
> called by
```
### System information
**The contents of your `packages.yml` file:**
This project uses `dependencies.yml`:
```yaml
packages:
- package: dbt-labs/dbt_utils
version: 1.4.1
- package: dbt-labs/codegen
version: 0.14.1
- package: godatadriven/dbt_date
version: 0.19.0
- package: dbt-labs/dbt_project_evaluator
version: 1.3.2
- package: brooklyn-data/dbt_artifacts
version: 2.10.1
- package: metaplane/dbt_expectations
version: 0.10.10
- package: starburstdata/trino_utils
version: 0.6.0
- package: Matts52/dbt_orphan
version: 0.1.3
```
**Which database are you using dbt with?**
* [ ] postgres
* [ ] redshift
* [ ] bigquery
* [ ] snowflake
* [x] other — **Trino**
The failure occurs during dbt name resolution, before any SQL is submitted to the database, so the issue does not appear to be adapter-specific.
**The output of `dbt --version`:**
```text
Core:
- installed: 1.12.0
- latest: 1.12.0 - Up to date!
Plugins:
- trino: 1.10.3 - Up to date!
```
**The operating system you're using:**
Ubuntu 24.04.4 LTS running under WSL2:
```text
6.18.33.2-microsoft-standard-WSL2
```
**The output of `python --version`:**
```text
Python 3.12.3
```
### Additional context
There are two places in `macros/generate_unit_test_template.sql` where the macro uses `item_dict.identifier`.
#### Source-column lookup
The first occurrence causes the compilation error:
```jinja
{%- if item_dict.resource_type == 'source' %}
{%- set columns = adapter.get_columns_in_relation(source(item_dict.source_name, item_dict.identifier)) -%}
```
#### Generated unit test input
The second occurrence writes an invalid source reference into the generated YAML:
```jinja
{% if item_dict.resource_type == 'source' %}
- input: source("{{item_dict.source_name}}", "{{item_dict.identifier}}")
```
The second occurrence is independently important. Even if the column lookup were fixed, the generated unit test would still contain:
```jinja
source("epicor_erp", "erp__part")
```
The unit test would then fail when it was executed.
`codegen.get_resource_from_unique_id` returns the source object from `graph.sources`, which already contains both `name` and `identifier`:
```jinja
{% macro get_resource_from_unique_id(resource_unique_id) %}
{% set resource_type = resource_unique_id.split('.')[0] %}
{% if resource_type == 'source' %}
{% set resource = graph.sources[resource_unique_id] %}
```
This means `item_dict.name` is already available at both affected call sites.
### Suggested fix
Replace `identifier` with `name` in both locations:
```diff
--- macros/generate_unit_test_template.sql
+++ macros/generate_unit_test_template.sql
@@ -24,7 +24,7 @@
{%- set input_columns_list = [] -%}
{%- set item_dict = codegen.get_resource_from_unique_id(item) -%}
{%- if item_dict.resource_type == 'source' %}
- {%- set columns = adapter.get_columns_in_relation(source(item_dict.source_name, item_dict.identifier)) -%}
+ {%- set columns = adapter.get_columns_in_relation(source(item_dict.source_name, item_dict.name)) -%}
{%- else -%}
{%- set columns = adapter.get_columns_in_relation(ref(item_dict.alias)) -%}
{%- endif -%}
@@ -59,7 +59,7 @@
{%- for i in range(ns.depends_on_list|length) -%}
{%- set item_dict = codegen.get_resource_from_unique_id(ns.depends_on_list[i]) -%}
{% if item_dict.resource_type == 'source' %}
- - input: source("{{item_dict.source_name}}", "{{item_dict.identifier}}")
+ - input: source("{{item_dict.source_name}}", "{{item_dict.name}}")
rows:
{%- else %}
- input: ref("{{item_dict.alias}}")
```
This change does not affect sources that do not override `identifier`, because `name` and `identifier` are already equal for those sources.
### Local verification
I tested this exact patch locally by overriding `default__generate_unit_test_template` within the project.
Apart from a header comment, the override was byte-for-byte identical to the upstream macro with only the two lines above changed.
After applying the patch, the same command that previously failed completed successfully and generated the following output:
```yaml
unit_tests:
- name: unit_test_base_epicor__part
model: base_epicor__part
given:
- input: source("epicor_erp", "part")
rows:
- company:
part_num:
search_word:
...
```
The generated `given` block now correctly references:
```jinja
source("epicor_erp", "part")
```
The operation also populated all 300 source columns and 240 model columns as expected.
### Workaround
Both affected lines are still present on `main` as of this report, so upgrading the package does not currently resolve the issue.
The available workaround is to override the macro in the project using `dispatch` in `dbt_project.yml`:
```yaml
dispatch:
- macro_namespace: codegen
search_order: ['', 'codegen']
```
Then copy `default__generate_unit_test_template` into the project and replace the two uses of `item_dict.identifier` with `item_dict.name`.
Contributor guide
Research direction
Start in macros/generate_unit_test_template.sql and inspect the two source branches that use item_dict.identifier. Run dbt run-operation generate_unit_test_template with the issue's model and source configuration; done means compilation succeeds and the generated YAML references source("epicor_erp", "part") while populating the columns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- testing-qa, tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100