ClickHouse / ClickHouse/dbt-clickhouse
dbt retry + microbatch fails to retry only failed batches
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
### Describe the bug
### Steps to reproduce
create the following base data (input_model):
```sql
{{
config(
materialized='table',
event_time='event_time'
)
}}
select 1 as id, toDateTime('2020-01-01 00:00:00') as event_time
union all
select 2 as id, toDateTime('2020-01-02 00:00:00') as event_time
union all
select 3 as id, toDateTime('2020-01-03 00:00:00') as event_time
```
And a model transforming the previous data (_microbatch_model_sql)
```sql
{{
config(
materialized='incremental',
incremental_strategy='microbatch',
unique_key='id',
event_time='event_time',
batch_size='day',
begin=modules.datetime.datetime(2020, 1, 1, 0, 0, 0)
)
}}
select * from {{ ref('input_model') }}
{% if model.batch.id == '20200103' %}
MONKEY TESTING INVALID SQL;
{% endif %}
```
`dbt run --event-time-start "2020-01-01" --event-time-end "2020-01-03"`
--> fails on batch 3. Let's fix it (_microbatch_model_sql_fixed)
```sql
{{
config(
materialized='incremental',
incremental_strategy='microbatch',
unique_key='id',
event_time='event_time',
batch_size='day',
begin=modules.datetime.datetime(2020, 1, 1, 0, 0, 0)
)
}}
select * from {{ ref('input_model') }}
```
`dbt retry` --> starts from first batch instead of third
### Expected behaviour
issuing `dbt retry` after the first fail should run the third batch only.
### Code examples, such as models or profile settings
### dbt and/or ClickHouse server logs
### Configuration
#### Environment
* dbt version: 1.9
* dbt-clickhouse version: 1.9.1
* clickhouse-driver version (if using native)
* clickhouse-connect version (if using http):
* Python version: 3.11
* Operating system: ubuntu
#### ClickHouse server
* ClickHouse Server version: 25.5.2.47
* ClickHouse Server non-default settings, if any:
* `CREATE TABLE` statements for tables involved:
Mmh see above
* Sample data for these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary
Contributor guide
Research direction
Reproduce the failure with the provided input_model and _microbatch_model_sql examples, then compare dbt run with dbt retry after removing the invalid SQL. Trace the retry and microbatch handling; done when retry executes only the failed third batch for the stated dbt-clickhouse and ClickHouse versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100