dbt-labs / dbt-labs/dbt

ClickHouse: implement source freshness (ClickHouseMetadataAdapter::freshness_inner)

Open
#14,653 0 comments 0 reactions 0 assignees View on GitHub
adapter:clickhouse area:adapters engine:v2 type:feature
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

## Context

`dbt source freshness` panics for ClickHouse with:

```
not yet implemented: ClickHouseAdapter::freshness
--> fs/sa/crates/dbt-adapter/src/metadata/clickhouse/mod.rs:263
```

The `freshness_inner` method on `ClickHouseMetadataAdapter` is a `todo!()`:

```rust
fn freshness_inner(
&self,
_relations: &[Arc],
_token: CancellationToken,
) -> AsyncAdapterResult<'_, BTreeMap> {
todo!("ClickHouseAdapter::freshness")
}
```

This means any project that configures source freshness on a ClickHouse source will hard-crash the engine.

## What's needed

Implement `freshness_inner` for ClickHouse — query each relation's last-modified timestamp and return a `MetadataFreshness` per relation. ClickHouse exposes this via `system.parts` or `system.tables` (the `metadata_modification_time` column).

## Location

`fs/sa/crates/dbt-adapter/src/metadata/clickhouse/mod.rs` — `freshness_inner` method.

## Tracking

The `adapters_clickhouse_metadata_source_freshness` test exists in `crates/dbt-adapter-tests/tests/adapter/clickhouse.rs` and is `#[ignore]`'d with this issue as the blocker. Un-ignore and record once implemented.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.