dbt-labs / dbt-labs/dbt

[Feature] microbatch_size should not be aligned to partition_by

Open
#11,278 7 comments 6 reactions 0 assignees View on GitHub
engine:v1 microbatch status:triage type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this your first time submitting a feature request?

- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

My upstream data changed & I want to be able to run a large backfill - 8 years worth.
This could be a common occurrence, so I want to set a project variable to change on job run whether the batch size is daily or monthly.

For bigquery, today I get a config error:
```
Exception on worker thread. Compilation Error in model
The 'microbatch' strategy requires a `partition_by` config with the same granularity as its configured `batch_size`.
Got:
`batch_size`: month
`partition_by.granularity`: day
```

Some incremental strategies don't need this (copy_partitions for sure).
Should this error only occur for merge strategies?

### Describe alternatives you've considered

Just run it foe each day - but its wasted time & compute

### Who will this benefit?

Users of mcirobatch - specifically BQ adapter

### Are you interested in contributing this feature?

Sure

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.