dbt-labs / dbt-labs/dbt

[CT-530] [Bug] Partial parsing should handle volatile variables like `invocation_id` and `run_started_at`

Open
#4,364 13 comments 0 reactions 0 assignees View on GitHub
engine:v1 Impact: Orch jira partial_parsing type:bug
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

You can trick dbt into giving dynamic freshness check behaviour with something like this:
```yml
freshness:
error_after:
count: "{{ 48 if run_started_at.strftime('%w') == '1' else 24 }}" #looser expectations on Monday
period: hour
```
[Community Slack thread](https://getdbt.slack.com/archives/C2JRTGLLS/p1638216863082300).

@jtcohen6 pointed out that
>Small word of warning — this won't be re-parsed by partial parsing, and so it's liable to get stale. Shouldn't be an issue in scheduled runs (which don't persist partial-parse state between runs), it might just be cause for surprise if you come back to it locally a few days from now.

https://docs.getdbt.com/reference/parsing#known-limitations

### Expected Behavior

Again from jerco:
>In the future, we should think about ways to always re-parse (invalidate cache) for any configs/properties/nodes/etc that depend on "volatile" variables like run_started_at or invocation_id

### Steps To Reproduce

1. Configure a source freshness check as described above, such that it will fail today and pass every other day of the week
2. Run `dbt source freshness`, note that the test fails
3. Wait a day (or change the code above to be dependent on the minute of the day or something)
4. Run `dbt source freshness` again. Note that the test still fails, even though it's a different day of the week.
5. Run `dbt source freshness --no-partial-parse`. Note that the test now passes, because it's re-parsed the project and re-evaluated `run_started_at`.

### Relevant log output

_No response_

### Environment

```markdown
- OS: macOS
- Python: 3.8.x
- dbt: 1.0-rc1
```

### What database are you using dbt with?

snowflake

### Additional Context

Related: https://github.com/dbt-labs/dbt-core/issues/2330

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.