dagster-io / dagster-io/dagster

Can we configure Automaterialization policies to start 15 minute after the table got updated?

Open
#17,543 0 comments 1 reaction 0 assignees View on GitHub
area: declarative-automation
Dominant language
Python
Stars
16.2k
Forks
2.3k
Avg merge
11d 1h
Merged PRs (30d)
3

Description

### Discussed in https://github.com/dagster-io/dagster/discussions/17541

Originally posted by **marcilj** October 31, 2023
I'm trying to use Auto materialization feature and I can't find any combinaison that work for my usecase.

Based on the [documentation](https://docs.dagster.io/concepts/assets/asset-auto-execution#auto-materialize-policies), the materialization options are:
```
AutoMaterializeRule.materialize_on_parent_updated
AutoMaterializeRule.materialize_on_missing
AutoMaterializeRule.materialize_on_required_for_freshness
AutoMaterializeRule.skip_on_parent_missing
AutoMaterializeRule.skip_on_parent_outdated
AutoMaterializeRule.skip_on_not_all_parents_updated
```

The problem I have is that I want to refresh my assets, 15 minute after `one` of it's parent has new records.

The reason behind that is that some of the parent aren't always getting refreshed. And since this is non-deterministic, I can't rely on a time base event to give me that info.

What I want to look at `AutoMaterializeRule.materialize_on_parent_updated` + `AutoMaterializeRule.skip_on_not_all_parents_updated` + `AutoMaterializeRule.materialize_on_required_for_freshness`.

But have `AutoMaterializeRule.materialize_on_required_for_freshness` set to start after the first event of ``AutoMaterializeRule.materialize_on_parent_updated` and not based on time.

My table might get refresh every 3 days, so if I setup a freshness policy of let's say 30 minutes, I'll refresh the table 144 times (24 hours * 2 * 3 days) instead of once 30 minutes after I receive any pieces of data.

Is there any way I can achieve that?

TLDR; I want my freshness policy to get activated by this event `AutoMaterializeRule.materialize_on_parent_updated`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.