aws-samples / aws-samples/dbt-glue

microbatch incremental model support

Open
#591 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
147
Forks
96
Avg merge
7h 4m
Merged PRs (30d)
5

Description

### Describe the feature
Add support for microbatch incremental model in dbt-glue adapter.
https://docs.getdbt.com/docs/build/incremental-microbatch

### Describe alternatives you've considered
Have considered merge and insert_overwrite but this does not scale well for large backfill operations.

### Additional context
get the below error when running a model with microbatch setting:

20:49:18 Unhandled error while executing
Exception on worker thread. Compilation Error in model stg_incremental_microbatch_test (models/stg_incremental_microbatch_test.sql)
Invalid incremental strategy provided: microbatch
Expected one of: 'append', 'merge', 'insert_overwrite'

### Who will this benefit?
This will benefit processing event data in small increments and to process backfills efficiently by refreshing subset of data at a time.

### Are you interested in contributing this feature?
Let us know if you want to write some code, and how we can help.

Contributor guide

Open the contributing guide

Research direction

Read the linked dbt microbatch documentation and reproduce the error with models/stg_incremental_microbatch_test.sql. Trace where the adapter validates incremental strategies, then verify that a microbatch-configured model compiles and processes increments and backfills without the shown invalid-strategy error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.