ClickHouse / ClickHouse/dbt-clickhouse

dbt retry + microbatch fails to retry only failed batches

Open
#482 0 comments 2 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

### Describe the bug

### Steps to reproduce

create the following base data (input_model):

```sql
{{
config(
materialized='table',
event_time='event_time'
)
}}
select 1 as id, toDateTime('2020-01-01 00:00:00') as event_time
union all
select 2 as id, toDateTime('2020-01-02 00:00:00') as event_time
union all
select 3 as id, toDateTime('2020-01-03 00:00:00') as event_time
```

And a model transforming the previous data (_microbatch_model_sql)

```sql
{{
config(
materialized='incremental',
incremental_strategy='microbatch',
unique_key='id',
event_time='event_time',
batch_size='day',
begin=modules.datetime.datetime(2020, 1, 1, 0, 0, 0)
)
}}
select * from {{ ref('input_model') }}

{% if model.batch.id == '20200103' %}
MONKEY TESTING INVALID SQL;
{% endif %}
```

`dbt run --event-time-start "2020-01-01" --event-time-end "2020-01-03"`
--> fails on batch 3. Let's fix it (_microbatch_model_sql_fixed)

```sql
{{
config(
materialized='incremental',
incremental_strategy='microbatch',
unique_key='id',
event_time='event_time',
batch_size='day',
begin=modules.datetime.datetime(2020, 1, 1, 0, 0, 0)
)
}}
select * from {{ ref('input_model') }}
```

`dbt retry` --> starts from first batch instead of third

### Expected behaviour

issuing `dbt retry` after the first fail should run the third batch only.

### Code examples, such as models or profile settings

### dbt and/or ClickHouse server logs

### Configuration
#### Environment
* dbt version: 1.9
* dbt-clickhouse version: 1.9.1
* clickhouse-driver version (if using native)
* clickhouse-connect version (if using http):
* Python version: 3.11
* Operating system: ubuntu

#### ClickHouse server
* ClickHouse Server version: 25.5.2.47
* ClickHouse Server non-default settings, if any:
* `CREATE TABLE` statements for tables involved:
Mmh see above
* Sample data for these tables, use [clickhouse-obfuscator](https://github.com/ClickHouse/ClickHouse/blob/master/programs/obfuscator/Obfuscator.cpp#L42-L80) if necessary

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the provided input_model and _microbatch_model_sql examples, then compare dbt run with dbt retry after removing the invalid SQL. Trace the retry and microbatch handling; done when retry executes only the failed third batch for the stated dbt-clickhouse and ClickHouse versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.