dbt-labs / dbt-labs/dbt

[Feature] dbt microbatch, alternative to now()-lookback to better ensure no data gaps

Open
#11,129 2 comments 5 reactions 0 assignees View on GitHub
discussion engine:v1 microbatch Refinement type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

# description

Currently for microbatch incremental strategy only way to handle latency between now() and real loaded time of data is to use `lookback` attribute in config.

Assuming there is model with `microbatch` strategy and `batch_size=day & lookback=3` and we load dbt run --full-refresh on `2024-10-07` (all batches till now()) once, and in subsequent days we load only batches for `now()-lookback` -> that will not cover cases where:
I. Latency for some batch is unexpectedly greater than usual. (ref. D on pic attached)
II. There is a gap in `dbt run` time (ref. G, H on pic attached)

![image](https://github.com/user-attachments/assets/1e456ce7-72be-43f9-8459-4abd73153f0f)

Problem arises also when:
III. There is model getting data from few `refs` , each one could have different latency

# proposed solution

introduce new attribute for `config` -> `max_event_time:True/False'`, default value `False`.

before running batches it will get max event times from all sources tables and take `min` of them. Calculates `min_of_max_event_time` parameter.

Then it will run batches between calculated `min_of_max_event_time-lookback` and `now()`

So `lookback` attribute could be used in both cases when config `max_event_time` is set `True` and `False`

Note. https://github.com/dbt-labs/dbt-core/issues/10702 is different because it is about only first run (`begin`)

### Describe alternatives you've considered

one workaround is to increase lookback , but in most of the time be waste of time and resources and can't be always 100% accurate

second is to create custom test and if missed `event_time` is detected run it using `--event-time-start` & `--event-time-end` flags. This introduce though troublesome additional maintenance time.

### Who will this benefit?

Teams which use big tables with random latency time for loading data & they want minimize maintenance time.

### Are you interested in contributing this feature?

yes

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.