dbt-labs / dbt-labs/dbt

feat(clickhouse): materialized_view materialization

Open
#14,587 1 comment 0 reactions 0 assignees View on GitHub
adapter:clickhouse area:adapters engine:v2
Dominant language
Rust
Stars
13.8k
Forks
2.6k
Avg merge
21h 31m
Merged PRs (30d)
56

Description

## Summary

The vendored `dbt-clickhouse` macro package includes a `materialized_view` materialization, but it is not ready to call supported in dbt-fusion yet.

This issue tracks both the direct Fusion compatibility blockers and the upstream dbt-clickhouse materialization cleanup needed before the Fusion port should be considered complete.

## Scope decision

Fusion should not blindly port all current dbt-clickhouse MV behavior until the shape is clarified.

The current Core adapter has two materialized-view modes:

- legacy mode: one dbt model creates a target table and one or more materialized views by parsing `-- name:begin` / `-- name:end` blocks out of the model SQL body
- external-target mode from ClickHouse/dbt-clickhouse#582: target tables and MVs are separate dbt models, with the MV target declared through `materialization_target_table(ref(...))`

The external-target shape is closer to the Fusion-friendly direction because it supports one dbt resource per ClickHouse resource. The open question is whether Fusion must also support the legacy multi-MV-in-one-model behavior or can defer/deprecate it.

## Known blockers

### 1. `modules.re` is not available in minijinja

`materialized_view.sql` currently uses Python regex from dbt's Jinja context:

- `modules.re.search(...)` to extract `materialization_target_table` from a generated SQL comment
- `modules.re.findall(...)` to discover and extract legacy multi-MV blocks

minijinja does not expose `modules.re`, so any model that relies on these paths will fail at render time.

Resolution options:

- refactor dbt-clickhouse so this metadata is not parsed from SQL/comments
- expose a narrow Fusion-compatible regex helper only if unavoidable
- move parsing to a Rust adapter layer with explicit tests
- gate unsupported legacy syntax with a clear error

### 2. SQL comment markers must survive normalization

The rendered SQL from `materialized_view.sql` contains special comment markers:

- `-- end_of_sql`
- `-- settings_section`
- `-- materialization_target_table: ...`
- legacy `-- name:begin` / `-- name:end` markers

These are used as control-plane delimiters, not normal SQL comments. The SQL normalizer must not strip them if Fusion continues to depend on them.

Preferred resolution: remove or reduce the need for magic comments by moving intent into config/metadata.

### 3. Upstream dbt-clickhouse MV contract needs clarification

Before Fusion can call dbt-labs/dbt-core#14587 complete, we need a clear answer for the dbt-clickhouse MV contract introduced in ClickHouse/dbt-clickhouse#582:

- Does Fusion support multiple MVs in one model?
- Is `materialization_target_table(ref(...))` the long-term API, or should target table selection become config/metadata?
- Which settings belong on the target table versus the MV?
- How should `mv_on_schema_change`, `catchup`, refreshable MV settings, and `repopulate_from_mvs_on_full_refresh` be represented?
- Can we keep model SQL focused on the query and move as much behavior as possible into `config()` or structured adapter APIs?

Upstream cleanup is tracked in ClickHouse/dbt-clickhouse#644.

## Work items

- [ ] Resolve or explicitly gate `modules.re` usage in `materialized_view.sql`
- [ ] Decide whether Fusion supports legacy multi-MV-in-one-model syntax
- [ ] Decide the Fusion-supported target-table API for MVs
- [ ] Prefer config/metadata over SQL comments for MV control-plane settings
- [ ] Ensure any required SQL comment markers survive normalization, or remove the dependency on them
- [ ] Add `materialized_view` to the supported ClickHouse materializations only after the API decision is made
- [ ] Record adapter tests covering external-target MV create, replace, full-refresh, and drop
- [ ] Add coverage for whatever legacy behavior is supported or a clear unsupported-feature error if it is intentionally not supported

## References

- Upstream cleanup: ClickHouse/dbt-clickhouse#644
- MV v2 implementation: ClickHouse/dbt-clickhouse#582
- Macro package: dbt-labs/fs#9883
- Beta materializations (Distributed, Dictionary, S3): dbt-labs/dbt-core#14580
- Parent EPIC: dbt-labs/dbt-fusion#1679

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.