Bug: incremental_batch flag not propagated to middle/last batches in MicrobatchModelRunner.execute()
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Is this a new bug in dbt-core?
- [x] I believe this is a new bug in dbt-core
- [x] I have searched the existing issues, and I could not find an existing issue for this bug
### Current Behavior
In `MicrobatchModelRunner.execute()` (`core/dbt/task/run.py`), the `incremental_batch` parameter is only passed to `_submit_batch()` for the **first batch** (line 778):
```python
# First batch — incremental_batch IS passed
relation_exists = self.parent_task._submit_batch(
...
incremental_batch=self._is_incremental(model=model), # ✓
)
```
For **middle batches** (lines 785-794) and the **last batch** (lines 817-827), the parameter is omitted:
```python
# Middle batches — incremental_batch NOT passed
relation_exists = self.parent_task._submit_batch(
node=model,
adapter=self.adapter,
relation_exists=relation_exists,
batches=batches,
batch_idx=batch_idx,
batch_results=batch_results,
pool=self.pool,
skip=skip_batches,
# ✗ NO incremental_batch parameter
)
```
Since `_submit_batch` defaults `incremental_batch=True` (line 910), middle and last batches **always behave as incremental** — even during `--full-refresh` or initial runs where `_is_incremental()` returns `False`.
This causes `is_incremental()` and `should_full_refresh()` in the Jinja context to have incorrect values for batches 2+ during full refresh / first run.
### Expected Behavior
All batch submissions should receive the same `incremental_batch` value computed from `_is_incremental(model=model)`. During `--full-refresh` or initial runs, `is_incremental()` should return `False` for **all** batches, not just the first one.
### Steps To Reproduce
1. Create a microbatch model that uses `is_incremental()` to conditionally change behavior
2. Run with `--full-refresh` and multiple batches (date range spanning >1 batch period)
3. Observe that `is_incremental()` returns `False` for batch 1 but `True` for batches 2+
### Suggested Fix
Compute the flag once before the loop and pass it to all three `_submit_batch()` calls:
```python
is_incremental = self._is_incremental(model=model)
# First batch
self.parent_task._submit_batch(..., incremental_batch=is_incremental)
# Middle batches
self.parent_task._submit_batch(..., incremental_batch=is_incremental)
# Last batch
self.parent_task._submit_batch(..., incremental_batch=is_incremental)
```
### Practical Impact
Low severity in practice — microbatch models typically rely on dbt's automatic `event_time` window filtering rather than `is_incremental()`, so the incorrect Jinja context is usually a no-op. However, it would affect models that use `is_incremental()` for conditional logic beyond time filtering (e.g., toggling merge strategy, applying `incremental_predicates`, or conditional schema handling).
### Environment
- dbt-core: main branch (commit 7acf053cb)
- File: `core/dbt/task/run.py`, lines 785-794 and 817-827
### Additional Context
Related epic: #11292
This bug follows the "parameter not propagated through all code paths" pattern. The existing tests verify end-to-end outcomes (row counts) but not per-batch Jinja context values, which is why it hasn't been caught.
Contributor guide
Assessment
This issue has not been assessed yet.