dbt-labs / dbt-labs/dbt-adapters

[Bug] When unit test introspects (describes) input sources - it does not respect the `quoting` config

Open
#1,261 0 comments 0 reactions 0 assignees View on GitHub
feature:quoting feature:unit-tests triage:product type:bug
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this a new bug?

- [x] I believe this is a new bug
- [x] I have searched the existing issues, and I could not find an existing issue for this bug

### Which packages are affected?

- [ ] dbt-adapters
- [ ] dbt-tests-adapter
- [ ] dbt-athena
- [ ] dbt-athena-community
- [ ] dbt-bigquery
- [ ] dbt-postgres
- [ ] dbt-redshift
- [x] dbt-snowflake
- [ ] dbt-spark

### Current Behavior

As per title... if we have a source that has a quoting config, and unit test inputs are using that source, the `quoting` property (https://docs.getdbt.com/reference/resource-properties/quoting) is ignored.

Models that make use of the source are not affected and `{{ source('raw', 'src') }}` resolves correctly as per the quoting property set.

### Expected Behavior

Unit test introspection respects the `quoting` property.

### Steps To Reproduce

1. Create a Snowflake relation that is purposely lowercased:

```sql
create or replace table db.raw."lower_case_identifier" as select 1 c;
```

Image

2. Use that source in dbt:

```yaml
# dbt_project.yml
name: analytics
profile: sf
version: "1.0.0"

models:
analytics:
+materialized: table

# models/sources.yml
sources:
- name: raw
quoting:
database: false
schema: false
identifier: true
tables:
- name: lower_case_identifier
quoting:
database: false
schema: false
identifier: true

unit_tests:
- name: test_is_valid
model: foo
given:
- input: source('raw', 'lower_case_identifier')
rows:
- { c: 1 }
expect:
rows:
- { double: 2 }
```

```sql
-- models/foo.sql
select c * 2 as double from {{ source('raw', 'lower_case_identifier') }}
```

3. Run `foo` to see that `quoting` works as expected:

```sh
$ dbt --debug run -s foo
01:53:10 Registered adapter: snowflake=1.10.0
01:53:10 checksum: 265d66e81f90af29a7a28579c9b54d938c94f4a57c57be5f853267ae509b5fcb, vars: {}, profile: , target: , version: 1.10.8
01:53:10 Partial parsing enabled: 0 files deleted, 0 files added, 1 files changed.
01:53:10 Partial parsing: updated file: analytics://models/sources.yml
01:53:10 Wrote artifact WritableManifest to /Users/jeremy/git/dbt-basic/target/manifest.json
01:53:10 Wrote artifact SemanticManifest to /Users/jeremy/git/dbt-basic/target/semantic_manifest.json
01:53:10 Found 1 model, 1 source, 477 macros, 1 unit test
01:53:10
01:53:10 Concurrency: 1 threads (target='ci')
01:53:10
01:53:10 Acquiring new snowflake connection 'master'
01:53:10 Acquiring new snowflake connection 'list_db'
01:53:10 Using snowflake connection "list_db"
01:53:10 On list_db: show terse schemas in database db
limit 10000
/* {"app": "dbt", "dbt_version": "1.10.8", "profile_name": "sf", "target_name": "ci", "connection_name": "list_db"} */
01:53:10 Opening a new connection, currently in state init
01:53:11 SQL status: SUCCESS 55 in 1.185 seconds
01:53:11 Re-using an available connection from the pool (formerly list_db, now list_db_sch)
01:53:11 Using snowflake connection "list_db_sch"
01:53:11 On list_db_sch: show objects in db.sch
limit 10000

/* {"app": "dbt", "dbt_version": "1.10.8", "profile_name": "sf", "target_name": "ci", "connection_name": "list_db_sch"} */;
01:53:12 SQL status: SUCCESS 174 in 0.463 seconds
01:53:12 Began running node model.analytics.foo
01:53:12 1 of 1 START sql table model sch.foo ........................................... [RUN]
01:53:12 Re-using an available connection from the pool (formerly list_db_sch, now model.analytics.foo)
01:53:12 Began compiling node model.analytics.foo
01:53:12 Writing injected SQL for node "model.analytics.foo"
01:53:12 Began executing node model.analytics.foo
01:53:12 Writing runtime sql for node "model.analytics.foo"
01:53:12 Using snowflake connection "model.analytics.foo"
01:53:12 On model.analytics.foo: create or replace transient table db.sch.foo



as (select c * 2 as double from db.raw."lower_case_identifier"
)

/* {"app": "dbt", "dbt_version": "1.10.8", "profile_name": "sf", "target_name": "ci", "node_id": "model.analytics.foo"} */;
01:53:13 SQL status: SUCCESS 1 in 1.455 seconds
01:53:13 1 of 1 OK created sql table model sch.foo ...................................... [SUCCESS 1 in 1.50s]
```

Notice the quoted identifier:

```sql
select c * 2 as double from db.raw."lower_case_identifier"
^^^^^^^^^^^^^^^^^^^^^^^
```

4. Let's execute the unit test:

```sh
$ dbt --debug test
01:55:06 Registered adapter: snowflake=1.10.0
01:55:06 checksum: 265d66e81f90af29a7a28579c9b54d938c94f4a57c57be5f853267ae509b5fcb, vars: {}, profile: , target: , version: 1.10.8
01:55:06 Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
01:55:06 Partial parsing enabled, no changes found, skipping parsing
01:55:06 Wrote artifact WritableManifest to /Users/jeremy/git/dbt-basic/target/manifest.json
01:55:06 Wrote artifact SemanticManifest to /Users/jeremy/git/dbt-basic/target/semantic_manifest.json
01:55:06 Found 1 model, 1 source, 477 macros, 1 unit test
01:55:06
01:55:06 Concurrency: 1 threads (target='ci')
01:55:06
01:55:06 Acquiring new snowflake connection 'master'
01:55:06 Acquiring new snowflake connection 'list_db_sch'
01:55:06 Using snowflake connection "list_db_sch"
01:55:06 On list_db_sch: show objects in db.sch
limit 10000

/* {"app": "dbt", "dbt_version": "1.10.8", "profile_name": "sf", "target_name": "ci", "connection_name": "list_db_sch"} */;
01:55:06 Opening a new connection, currently in state init
01:55:07 SQL status: SUCCESS 174 in 1.226 seconds
01:55:07 Began running node unit_test.analytics.foo.test_is_valid
01:55:07 1 of 1 START unit_test foo::test_is_valid ...................................... [RUN]
01:55:07 Re-using an available connection from the pool (formerly list_db_sch, now unit_test.analytics.foo.test_is_valid)
01:55:07 Began compiling node unit_test.analytics.foo.test_is_valid
01:55:07 Began executing node unit_test.analytics.foo.test_is_valid
01:55:07 Using snowflake connection "unit_test.analytics.foo.test_is_valid"
01:55:07 On unit_test.analytics.foo.test_is_valid: describe table db.raw.lower_case_identifier
/* {"app": "dbt", "dbt_version": "1.10.8", "profile_name": "sf", "target_name": "ci", "node_id": "unit_test.analytics.foo.test_is_valid"} */
01:55:08 Snowflake adapter: Snowflake query id: 01be5513-0609-2cb6-000d-37835174a53a
01:55:08 Snowflake adapter: Snowflake error: 002003 (42S02): SQL compilation error:
Table 'DB.RAW.LOWER_CASE_IDENTIFIER' does not exist or not authorized.
01:55:08 Snowflake adapter: Error running SQL: macro get_columns_in_relation
01:55:08 Snowflake adapter: Rolling back transaction.
01:55:08 Compilation Error in model lower_case_identifier (models/sources.yml)
Not able to get columns for unit test 'lower_case_identifier' from relation db.raw.lower_case_identifier because the relation doesn't exist

> in macro get_fixture_sql (macros/unit_test_sql/get_fixture_sql.sql)
> called by model lower_case_identifier (models/sources.yml)
01:55:08 1 of 1 ERROR foo::test_is_valid ................................................ [ERROR in 0.37s]
01:55:08 Finished running node unit_test.analytics.foo.test_is_valid
01:55:08 Marking all children of 'unit_test.analytics.foo.test_is_valid' to be skipped because of status 'error'. Reason: Compilation Error in model lower_case_identifier (models/sources.yml)
Not able to get columns for unit test 'lower_case_identifier' from relation db.raw.lower_case_identifier because the relation doesn't exist

> in macro get_fixture_sql (macros/unit_test_sql/get_fixture_sql.sql)
> called by model lower_case_identifier (models/sources.yml).
01:55:08 Connection 'master' was properly closed.
01:55:08 Connection 'unit_test.analytics.foo.test_is_valid' was left open.
01:55:08 On unit_test.analytics.foo.test_is_valid: Close
01:55:08
01:55:08 Finished running 1 unit test in 0 hours 0 minutes and 2.08 seconds (2.08s).
01:55:08 Command end result
01:55:08 Wrote artifact WritableManifest to /Users/jeremy/git/dbt-basic/target/manifest.json
01:55:08 Wrote artifact SemanticManifest to /Users/jeremy/git/dbt-basic/target/semantic_manifest.json
01:55:08 Wrote artifact RunExecutionResult to /Users/jeremy/git/dbt-basic/target/run_results.json
01:55:08
01:55:08 Completed with 1 error, 0 partial successes, and 0 warnings:
01:55:08
01:55:08 Failure in unit_test test_is_valid (models/sources.yml)
01:55:08 Compilation Error in model lower_case_identifier (models/sources.yml)
Not able to get columns for unit test 'lower_case_identifier' from relation db.raw.lower_case_identifier because the relation doesn't exist

> in macro get_fixture_sql (macros/unit_test_sql/get_fixture_sql.sql)
> called by model lower_case_identifier (models/sources.yml)
01:55:08
01:55:08 Done. PASS=0 WARN=0 ERROR=1 SKIP=0 NO-OP=0 TOTAL=1
```

Notice the non quoted identifier when introspecting the source:

```sql
describe table db.raw.lower_case_identifier
^^^^^^^^^^^^^^^^^^^^^
```

### Relevant log output

```shell

```

### Environment

```markdown
- OS: macOS
- Python: 3.11.9
- dbt-adapters: 1.16.3
- dbt-snowflake: 1.10.0
- dbt-core: 1.10.8
```

### Additional Context

I believe we run introspection due to:

https://github.com/dbt-labs/dbt-adapters/blob/6dc698497b4123813210f332b09f04960b0a04f7/dbt-adapters/src/dbt/include/global_project/macros/unit_test_sql/get_fixture_sql.sql#L8

However, not sure why that would skip the quoting config. Doing a quick test with that method:

```sql
--macros/check.sql
{% macro check() %}
{% do adapter.get_columns_in_relation(source('raw', 'lower_case_identifier')) %}
{% endmacro %}
```

```sh
$ dbt --debug run-operation check
02:03:32 Found 1 model, 1 source, 478 macros, 1 unit test
02:03:32 Acquiring new snowflake connection 'macro_check'
02:03:32 Using snowflake connection "macro_check"
02:03:32 On macro_check: describe table db.raw."lower_case_identifier"
/* {"app": "dbt", "dbt_version": "1.10.8", "profile_name": "sf", "target_name": "ci", "connection_name": "macro_check"} */
02:03:32 Opening a new connection, currently in state init
02:03:33 SQL status: SUCCESS 1 in 0.980 seconds
02:03:33 Wrote artifact RunResultsArtifact to /Users/jeremy/git/dbt-basic/target/run_results.json
02:03:33 Resource report: {"command_name": "run-operation", "command_success": true, "command_wall_clock_time": 1.6225426, "process_in_blocks": "0", "process_kernel_time": 0.178157, "process_mem_max_rss": "215597056", "process_out_blocks": "0", "process_user_time": 1.138828}
02:03:33 Command `dbt run-operation` succeeded at 14:03:33.097642 after 1.62 seconds
```

We're definitely applying the right quoting here...

I think it's something to do with the fact that we're using the special `this` variable? Modifying the ootb macro slightly:

```sql
{% macro get_fixture_sql(rows, column_name_to_data_types) %}
...
{%- if not column_name_to_data_types -%}
{#-- Use defer_relation IFF it is available in the manifest and 'this' is missing from the database --#}
{%- set this_or_defer_relation = defer_relation if (defer_relation and not load_relation(this)) else this -%}
{% do log('>>>>>>>>>>') %}
{% do log(this) %}
{% do log('>>>>>>>>>>') %}
{%- set columns_in_relation = adapter.get_columns_in_relation(this_or_defer_relation) -%}
...
{% endmacro %}
```

```sh
$ dbt --debug test
...
02:13:59 1 of 1 START unit_test foo::test_is_valid ...................................... [RUN]
02:13:59 Re-using an available connection from the pool (formerly list_db_sch, now unit_test.analytics.foo.test_is_valid)
02:13:59 Began compiling node unit_test.analytics.foo.test_is_valid
02:13:59 Began executing node unit_test.analytics.foo.test_is_valid
02:14:00 >>>>>>>>>>
02:14:00 db.raw.lower_case_identifier
02:14:00 >>>>>>>>>>
02:14:00 Using snowflake connection "unit_test.analytics.foo.test_is_valid"
...
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.