dbt-labs / dbt-labs/dbt

[v2 Bug] dbt Fusion generates invalid SQL for unit tests on incremental models with nested not_null constraints (BigQuery)

Open
#15,878 0 comments 0 reactions 0 assignees View on GitHub
area:engine status:triage type:bug
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this a new bug in dbt v2.x compared to the latest version of dbt 1.x?

- [x] I believe this is a new bug in dbt v2.x
- [x] I have searched the existing issues and could not find a duplicate

### Current Behavior

When a model has an enforced contract, is materialized as `incremental`, and declares a `not_null` constraint on a nested field inside an `ARRAY>` column (using dot notation, e.g. `my_array.sub_id`), running a unit test against that model with dbt Fusion fails with a BigQuery syntax error.

The generated mock/expected result query builds the expected row using a `CAST(... AS ARRAY>)` expression. dbt Fusion includes `not null` inside that nested `STRUCT` type declaration, which BigQuery does not accept inside a `CAST` type expression (as opposed to a `CREATE TABLE` column definition, where it is valid). This produces a syntax error and the unit test fails to run at all.

This only happens when the model is materialized as `incremental`. The same model materialized as `table` runs the unit test successfully.

### Expected Behavior

Running a unit test with dbt Fusion should produce valid SQL regardless of the model's materialization. The `CAST` expression used to build the mock/expected result set should not include constraint modifiers such as `not null`, since those only apply in a `CREATE TABLE` column definition context. The unit test should pass or fail based on the actual row comparison, not fail with a database syntax error.

### Steps To Reproduce

1. Create the following model:

`models/required_nested.sql`
```sql
SELECT
1 as id,
[
STRUCT(
'element_1' AS sub_id,
STRUCT(
'required1' AS first_field,
CAST(NULL AS STRING) AS second_field
) AS my_struct
)
] AS my_array
```

`models/required_nested.yml`
```yaml
version: 2

models:
- name: required_nested
config:
contract:
enforced: true
materialized: incremental
incremental_strategy: merge
unique_key: id
on_schema_change: append_new_columns
columns:
- name: id
data_type: int64
constraints:
- type: not_null
- type: primary_key
data_tests:
- unique
- name: my_array
data_type: array
- name: my_array.sub_id
data_type: string
constraints:
- type: not_null
- name: my_array.my_struct
data_type: struct
- name: my_array.my_struct.first_field
data_type: string
constraints:
- type: not_null
- name: my_array.my_struct.second_field
data_type: string

unit_tests:
- name: test__required_nested
model: required_nested
given: []
expect:
rows:
- id: 1
my_array: ['STRUCT("element_1" AS sub_id, STRUCT("required1" AS first_field, CAST(NULL AS STRING) AS second_field) AS my_struct)']
```

2. Make sure the underlying table already exists in BigQuery with the nested fields correctly set as `REQUIRED` (for example, by building the same model once with dbt Core / the dbt Cloud CLI).

3. Run the unit test with dbt Fusion:

```
dbtf test --select test__required_nested
```

4. Observe the syntax error below.

### Relevant log output

```shell
Compiled query used to build the expected row for the unit test:

CAST(
[STRUCT("element_1" AS sub_id, STRUCT("required1" AS first_field, CAST(NULL AS STRING) AS second_field) AS my_struct)]
AS ARRAY>>
)

Resulting error:

[error] [JinjaError (dbt1501)]: Error materializing unit test unit_test.my_project.required_nested.test__required_nested: [BigQuery] googleapi: Error 400: Syntax error: Expected "," or ">" but got keyword NOT at [4:289], invalidQuery
(in run/my_project/models/test__required_nested.sql:2:32)
--> target/compiled/my_project/models/test__required_nested.sql
```

### Environment

```markdown
- OS:
- CPU: (x86 or ARM)
- dbt distribution and version: (`dbt --version`) 2.0.0-preview.205
```

### Which database adapter are you using?

bigquery

### Is this a discrepancy vs. dbt 1.x?

- [x] Yes — this works in dbt 1.x but not in dbt v2.x

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.