dbt-labs / dbt-labs/dbt

[Feature] Microbatch - Lookback vs lag

Open
#15,518 0 comments 0 reactions 0 assignees View on GitHub
triage type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Which version of dbt is this feature for?

Not sure / both

### Describe the feature

I think we are missing quite crucial aspect of microbatch. What a merge incremental model still does better is that if there's a gap in processing of data, it will backfill automatically due to MAX(timestamp) approach.

Microbatch doesn't have any awareness of it. Lookback is for reprocessing the data that you know needs to be reprocessed like late-arriving data or simply the time when you run the job.

It should, however, also have a setting that simply checks if batches exists and if not, it runs them too..

### Describe alternatives you've considered

_No response_

### Who will this benefit?

_No response_

### Are you interested in contributing this feature?

_No response_

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.