dbt-labs / dbt-labs/dbt

[Feature] Could micro-batch strategy solve most of my windowing problems for incremental models?

Open
#11,059 0 comments 0 reactions 0 assignees View on GitHub
engine:v1 status:triage type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Microbatch should handle windowing logic.

But the hardest step for any incremental model has been figuring out what to do with windowing logic - and still having performant model pipelines.

Microbatch is fantastic! For an org with large-ish datasets, the feature makes moving data easy and straightforward.

### Describe alternatives you've considered

Should there just be a blog post explaining how users can custom fit their microbatch strategies?

Here's an example where the microbatch model is customized to
- Get a 15 day window for all days if the date is within the past 2 weeks. This handles late arriving date.
- Get a 7 day window for 1 day a week if the date is older than 2 weeks ago.

```
{# Custom microbatch selection for windowing #}
{# Get the starttime for a microbatch & a window up to 15 days ago #}
{%- set microbatch_day -%}
{{ model.config.__dbt_internal_microbatch_event_time_start or modules.datetime.date(1899, 1, 1) }}
{%- endset -%}
{% set fifteen_days_ago = (modules.datetime.date.today() - modules.datetime.timedelta(days=15)) %}
{% set only_run_mondays %}
and EXTRACT(dayofweek FROM DATE('{{ microbatch_day }}')) = 2
{% endset %}
with
filter_microbatch__user_daily_agent_id_active_days_tbl as (

select *
from {{ ref('user_daily_agent_id_active_days_tbl').render() }}
--override default microbatch behavior with 7 day window
-- This process the full week for each lookback window - which is wasteful.
where date(_pt) >= date_trunc(date('{{ microbatch_day }}'), week(monday)) - 7
and date(_pt) < date('{{ microbatch_day }}')
{# Performance improvement - filter to only run partition on 1 day of week if older than 15 days ago. #}
{{ only_run_mondays if fifteen_days_ago > modules.datetime.date.fromisoformat(microbatch_day) }}
)

select
{{ dbt_utils.generate_surrogate_key(['fiscal_week', 'agent_id']) }} as uid,
fiscal_week,
fiscal_week as _pt,
agent_id,
max(1) as is_active_this_week,
count(distinct _pt) as active_days,
SUM(is_active_on_news_today) as active_days_on_news
from filter_microbatch__user_daily_agent_id_active_days_tbl
group by all
```

### Who will this benefit?

The dbt community has been talking about [windowing and incremental models ](https://discourse.getdbt.com/t/using-window-functions-in-incremental-tables/81)since 2018 - and look Tristan even weighed in!

Any user of the microbatch strategy will probably benefit. Most models eventually mature to use window functions.

### Are you interested in contributing this feature?

Sure!

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.