dbt-labs / dbt-labs/docs.getdbt.com

Improve dbt documentation with real-world examples for incremental models (Snowflake-focused)

Open
#9,150 0 comments 0 reactions 0 assignees View on GitHub
content improvement
Dominant language
JavaScript
Stars
215
Forks
1.2k
Avg merge
1d 15h
Merged PRs (30d)
130

Description

### Contributions

- [x] I have read the contribution docs, and understand what's expected of me.

### Link to the page on docs.getdbt.com requiring updates

Hi team,

I’d like to propose an improvement to the dbt documentation around incremental models.

### Problem
The current documentation explains incremental models well conceptually, but lacks real-world production examples, especially for Snowflake users.

In practice, data engineers often need to handle:
- Deduplication during incremental loads
- Late-arriving data
- Merge-based strategies with update conditions
- Idempotent pipeline design

These scenarios are common in production pipelines but are not clearly demonstrated in the current docs.

### Proposed improvement
Add a section under incremental models with practical examples such as:
- Deduplication using ROW_NUMBER()
- Merge logic with conditional updates
- Handling late-arriving records using timestamps
- Example Snowflake-based incremental model

### Example snippet (simplified)

```sql
{{ config(materialized='incremental', unique_key='id') }}

with source as (
select *
from {{ source('raw', 'orders') }}
),

deduplicated as (
select *
from (
select *,
row_number() over (partition by id order by updated_at desc) as rn
from source
)
where rn = 1
)

select *
from deduplicated

{% if is_incremental() %}
where updated_at > (select max(updated_at) from {{ this }})
{% endif %}

### What part(s) of the page would you like to see updated?

The current incremental models documentation explains the concept and basic usage, but it lacks real-world production scenarios that data engineers commonly face.

Specifically, the following areas could be improved:

- There are no clear examples showing how to handle deduplication during incremental loads (e.g., using ROW_NUMBER or latest record selection).
- Late-arriving data handling is not demonstrated (e.g., records arriving after the initial load based on timestamps).
- Merge-based incremental strategies (especially for Snowflake) are not explained with practical SQL examples.
- There is limited guidance on building idempotent pipelines where reruns do not create duplicates or inconsistencies.

Why should the docs be changed:
In real-world data engineering pipelines, incremental models are rarely used in their simplest form. Engineers often need to combine filtering, deduplication, and merge logic to ensure data correctness. Without these examples, users may implement inefficient or incorrect patterns.

This improvement would support:
- Data engineers working with Snowflake and large-scale pipelines
- Teams implementing medallion architectures (RAW → staging → marts)
- Users trying to avoid duplication and maintain data consistency in incremental loads

Expected outcome:
- Add a new section or expand the existing incremental models page with practical, production-ready examples
- Include at least one Snowflake-based example demonstrating:
- Deduplication using window functions
- Incremental filtering using timestamps
- Merge logic with a unique key
- Improve clarity so users can directly apply these patterns in real-world pipelines

### Additional information

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.