dbt-labs / dbt-labs/dbt

Bug: incremental_batch flag not propagated to middle/last batches in MicrobatchModelRunner.execute()

Open
#12,682 0 comments 0 reactions 0 assignees View on GitHub
engine:v1
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

### Is this a new bug in dbt-core?

- [x] I believe this is a new bug in dbt-core
- [x] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

In `MicrobatchModelRunner.execute()` (`core/dbt/task/run.py`), the `incremental_batch` parameter is only passed to `_submit_batch()` for the **first batch** (line 778):

```python
# First batch — incremental_batch IS passed
relation_exists = self.parent_task._submit_batch(
...
incremental_batch=self._is_incremental(model=model), # ✓
)
```

For **middle batches** (lines 785-794) and the **last batch** (lines 817-827), the parameter is omitted:

```python
# Middle batches — incremental_batch NOT passed
relation_exists = self.parent_task._submit_batch(
node=model,
adapter=self.adapter,
relation_exists=relation_exists,
batches=batches,
batch_idx=batch_idx,
batch_results=batch_results,
pool=self.pool,
skip=skip_batches,
# ✗ NO incremental_batch parameter
)
```

Since `_submit_batch` defaults `incremental_batch=True` (line 910), middle and last batches **always behave as incremental** — even during `--full-refresh` or initial runs where `_is_incremental()` returns `False`.

This causes `is_incremental()` and `should_full_refresh()` in the Jinja context to have incorrect values for batches 2+ during full refresh / first run.

### Expected Behavior

All batch submissions should receive the same `incremental_batch` value computed from `_is_incremental(model=model)`. During `--full-refresh` or initial runs, `is_incremental()` should return `False` for **all** batches, not just the first one.

### Steps To Reproduce

1. Create a microbatch model that uses `is_incremental()` to conditionally change behavior
2. Run with `--full-refresh` and multiple batches (date range spanning >1 batch period)
3. Observe that `is_incremental()` returns `False` for batch 1 but `True` for batches 2+

### Suggested Fix

Compute the flag once before the loop and pass it to all three `_submit_batch()` calls:

```python
is_incremental = self._is_incremental(model=model)

# First batch
self.parent_task._submit_batch(..., incremental_batch=is_incremental)

# Middle batches
self.parent_task._submit_batch(..., incremental_batch=is_incremental)

# Last batch
self.parent_task._submit_batch(..., incremental_batch=is_incremental)
```

### Practical Impact

Low severity in practice — microbatch models typically rely on dbt's automatic `event_time` window filtering rather than `is_incremental()`, so the incorrect Jinja context is usually a no-op. However, it would affect models that use `is_incremental()` for conditional logic beyond time filtering (e.g., toggling merge strategy, applying `incremental_predicates`, or conditional schema handling).

### Environment

- dbt-core: main branch (commit 7acf053cb)
- File: `core/dbt/task/run.py`, lines 785-794 and 817-827

### Additional Context

Related epic: #11292

This bug follows the "parameter not propagated through all code paths" pattern. The existing tests verify end-to-end outcomes (row counts) but not per-batch Jinja context values, which is why it hasn't been caught.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.