aws-samples / aws-samples/dbt-glue
microbatch incremental model support
- Dominant language
- Python
- Stars
- 147
- Forks
- 96
- Avg merge
- 7h 4m
- Merged PRs (30d)
- 5
Description
### Describe the feature
Add support for microbatch incremental model in dbt-glue adapter.
https://docs.getdbt.com/docs/build/incremental-microbatch
### Describe alternatives you've considered
Have considered merge and insert_overwrite but this does not scale well for large backfill operations.
### Additional context
get the below error when running a model with microbatch setting:
20:49:18 Unhandled error while executing
Exception on worker thread. Compilation Error in model stg_incremental_microbatch_test (models/stg_incremental_microbatch_test.sql)
Invalid incremental strategy provided: microbatch
Expected one of: 'append', 'merge', 'insert_overwrite'
### Who will this benefit?
This will benefit processing event data in small increments and to process backfills efficiently by refreshing subset of data at a time.
### Are you interested in contributing this feature?
Let us know if you want to write some code, and how we can help.
Contributor guide
Research direction
Read the linked dbt microbatch documentation and reproduce the error with models/stg_incremental_microbatch_test.sql. Trace where the adapter validates incremental strategies, then verify that a microbatch-configured model compiles and processes increments and backfills without the shown invalid-strategy error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100