dbt-labs / dbt-labs/dbt

[Feature] Support automatic group inheritance in singular data test

Open
#12,107 1 comment 0 reactions 0 assignees View on GitHub
dbt tests engine:v1 status:triage type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

dbt currently supports:
1. Specifying a group.
2. Declaring that a model belongs to a group (a model can belong to only 1 group).

Additionally, when declaring model level generic test - those generic test automatically inherit the group of the model. Let's see this in action:

```yaml
# dbt_project.yml
name: analytics
profile: all
version: "1.0.0"
models:
analytics:
+materialized: table

# models/schema.yml
groups:
- name: finance
owner:
email: jeremy.yeo@dbtlabs.com

models:
- name: foo
config:
group: finance
columns:
- name: id
tests:
- not_null
```

```sql
-- models/foo.sql
select 1 id
```

Compile the project and inspect the `manifest.json`:

```json
{
"database": "db",
"schema": "sch",
"name": "foo",
"resource_type": "model",
"package_name": "analytics",
"path": "marts/foo.sql",
"original_file_path": "models/marts/foo.sql",
"unique_id": "model.analytics.foo",
...
"group": "finance",
...
}

{
"database": "db",
"schema": "sch_dbt_test__audit",
"name": "not_null_foo_id",
"resource_type": "test",
"package_name": "analytics",
"path": "not_null_foo_id.sql",
"original_file_path": "models/src.yml",
"unique_id": "test.analytics.not_null_foo_id.f099b1e59c",
...
"group": "finance",
...
}
```

We can see that the test node `test.analytics.not_null_foo_id` inherited the model's group. This is easy to reason about... logically, model `foo` has a group of "finance" and the test is applied directly on that model itself.

What happens if we use model `foo` in a singular data test?

```sql
-- test/my_data_test.sql
-- Ensure there is no 0 value.
select * from {{ ref('foo') }} where id = 0
```

```json
{
"database": "db",
"schema": "sch_dbt_test__audit",
"name": "my_data_test",
"resource_type": "test",
"package_name": "analytics",
"path": "my_data_test.sql",
"original_file_path": "tests/my_data_test.sql",
"unique_id": "test.analytics.my_data_test",
...
"group": null,
...
}
```

We can see that the group information is `null` - why is that? This is because singular data test are it's own unique type of node and any code written within it is does not have the same type of relationship as a straight forward "model is in a group, generic test on model, generic test inherits models group" - for example, lets say we have another model and group in our project:

```yaml
# dbt_project.yml
name: analytics
profile: all
version: "1.0.0"
models:
analytics:
+materialized: table

# models/schema.yml
groups:
- name: finance
owner:
email: jeremy.yeo@dbtlabs.com
- name: marketing
owner:
email: jeremy.yeo@dbtlabs.com

models:
- name: foo
config:
group: finance
columns:
- name: id
tests:
- not_null
- name: bar
config:
group: marketing
```

```sql
-- models/foo.sql
select 1 id

-- models/bar.sql
select 2 id
```

```sql
-- test/my_data_test.sql
select * from {{ ref('foo') }} where id not in (select id from {{ ref('bar') }})
```

Recall that a node (model, test) can only have 1 group - therefore what "group" would this `my_data_test` belong to - when `foo` belongs to `finance` and `bar` belongs to `marketing` and both of those models are being used in the same singular data test? It's not obvious which group to "pick" and you also cannot have more than 1 group per node. In the generic test scenario - ownership (group) of a model and therefore it's direct test is straight forward. But on a singular data test scenario - it is not - a singular data test is as if it is a "model" itself.

Opening up this FR to figure out a path forward in terms of automatic group inheritance for singular data tests.

### Describe alternatives you've considered

Users need to manually specify the `group` in the singular data test itself:

```sql
-- test/my_data_test.sql
{{ config(group='finance') }}
select * from {{ ref('foo') }} where id not in (select id from {{ ref('bar') }})
```

```json
{
"database": "db",
"schema": "sch_dbt_test__audit",
"name": "my_data_test",
"resource_type": "test",
"package_name": "analytics",
"path": "my_data_test.sql",
"original_file_path": "tests/my_data_test.sql",
"unique_id": "test.analytics.my_data_test",
...
"group": "finance",
...
}
```

^ However, that necessarily means that that singular data test will not be tagged with `"group": "marketing"`.

### Who will this benefit?

Model groups power model level notifications (https://docs.getdbt.com/docs/deploy/model-notifications) in dbt Platform and so this would be useful for those users.

### Are you interested in contributing this feature?

_No response_

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.