ClickHouse / ClickHouse/dbt-clickhouse

dbt snapshot - check strategy - stopped working properly

Open
#481 16 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

Snapshot check strategy odd behavior. While timestamp strategy works fine, snapshot check strategy works quite odd and works fine only if _dbt_updated_at_ parameter in .yml config is used (similar to timestamp strategy).
Main issue is with inconsistent results - historic version rows are not closed (they remain valid to 2100-01-01), new version rows are automatically closed (they are not valid to 2100-01-01), etc.

### Steps to reproduce
1. Create table
2. Run initial snapshot - everything works
3. Update some column, for example "company_name"
4. Run snapshot again - as stated above - old version rows are not closed, new version rows are automatically closed, etc.

### Expected behavior
dbt_valid_to = some date/time - ie. actual time of snapshot run - historic version
dbt_valid_to = 2100-01-01 00:00:00 for opened current versions

### Configuration
#### Environment
* dbt version: 1.9.4
* dbt-clickhouse version: 1.9.1

#### ClickHouse server
* ClickHouse Server version: 25.6.3.116

### Snapshot model
```
{% snapshot snp_test_table %}

{{ config(
query_settings = {'join_use_nulls': 1},
target_schema=generate_schema_name('snapshots')
)
}}

SELECT
{{ dbt_utils.star(
from = ref('stg_test_table')
) }}
FROM {{ ref('stg_test_table') }}

{% endsnapshot %}
```

### Snapshot yml
```
version: 2
snapshots:
- name: snp_test_table
description: 'Test table'
latest_version: 1
versions:
- v: 1
config:
alias: snp_test_table
config:
unique_key: vat_number
strategy: check
dbt_valid_to_current: "{{ var('future_proof_date') }}"
hard_deletes: 'invalidate'
check_cols:
- company_name
- post_code
- company_city
group: TGDA
tags:
- test_table
```

var('future_proof_date') is defined in the dbt_project.yml file as:
`vars:
future_proof_date: "toDateTime('2100-01-01 00:00:00')"`

#### DDL's

```
CREATE TABLE services.test_table
(

`vat_number` String,

`company_name` String,

`post_code` String,

`company_city` String
)
ENGINE = MergeTree
ORDER BY vat_number
SETTINGS index_granularity = 8192;
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the snapshot model and snapshot.yml configuration against dbt 1.9.4, dbt-clickhouse 1.9.1, and ClickHouse 25.6.3.116. Compare dbt_valid_to values after the initial run and an update to company_name; done means historic rows close at the snapshot time and current rows retain the configured 2100-01-01 value.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.