feat(clickhouse): materialized_view materialization
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
## Summary
The vendored `dbt-clickhouse` macro package includes a `materialized_view` materialization, but it is not ready to call supported in dbt-fusion yet.
This issue tracks both the direct Fusion compatibility blockers and the upstream dbt-clickhouse materialization cleanup needed before the Fusion port should be considered complete.
## Scope decision
Fusion should not blindly port all current dbt-clickhouse MV behavior until the shape is clarified.
The current Core adapter has two materialized-view modes:
- legacy mode: one dbt model creates a target table and one or more materialized views by parsing `-- name:begin` / `-- name:end` blocks out of the model SQL body
- external-target mode from ClickHouse/dbt-clickhouse#582: target tables and MVs are separate dbt models, with the MV target declared through `materialization_target_table(ref(...))`
The external-target shape is closer to the Fusion-friendly direction because it supports one dbt resource per ClickHouse resource. The open question is whether Fusion must also support the legacy multi-MV-in-one-model behavior or can defer/deprecate it.
## Known blockers
### 1. `modules.re` is not available in minijinja
`materialized_view.sql` currently uses Python regex from dbt's Jinja context:
- `modules.re.search(...)` to extract `materialization_target_table` from a generated SQL comment
- `modules.re.findall(...)` to discover and extract legacy multi-MV blocks
minijinja does not expose `modules.re`, so any model that relies on these paths will fail at render time.
Resolution options:
- refactor dbt-clickhouse so this metadata is not parsed from SQL/comments
- expose a narrow Fusion-compatible regex helper only if unavoidable
- move parsing to a Rust adapter layer with explicit tests
- gate unsupported legacy syntax with a clear error
### 2. SQL comment markers must survive normalization
The rendered SQL from `materialized_view.sql` contains special comment markers:
- `-- end_of_sql`
- `-- settings_section`
- `-- materialization_target_table: ...`
- legacy `-- name:begin` / `-- name:end` markers
These are used as control-plane delimiters, not normal SQL comments. The SQL normalizer must not strip them if Fusion continues to depend on them.
Preferred resolution: remove or reduce the need for magic comments by moving intent into config/metadata.
### 3. Upstream dbt-clickhouse MV contract needs clarification
Before Fusion can call dbt-labs/dbt-core#14587 complete, we need a clear answer for the dbt-clickhouse MV contract introduced in ClickHouse/dbt-clickhouse#582:
- Does Fusion support multiple MVs in one model?
- Is `materialization_target_table(ref(...))` the long-term API, or should target table selection become config/metadata?
- Which settings belong on the target table versus the MV?
- How should `mv_on_schema_change`, `catchup`, refreshable MV settings, and `repopulate_from_mvs_on_full_refresh` be represented?
- Can we keep model SQL focused on the query and move as much behavior as possible into `config()` or structured adapter APIs?
Upstream cleanup is tracked in ClickHouse/dbt-clickhouse#644.
## Work items
- [ ] Resolve or explicitly gate `modules.re` usage in `materialized_view.sql`
- [ ] Decide whether Fusion supports legacy multi-MV-in-one-model syntax
- [ ] Decide the Fusion-supported target-table API for MVs
- [ ] Prefer config/metadata over SQL comments for MV control-plane settings
- [ ] Ensure any required SQL comment markers survive normalization, or remove the dependency on them
- [ ] Add `materialized_view` to the supported ClickHouse materializations only after the API decision is made
- [ ] Record adapter tests covering external-target MV create, replace, full-refresh, and drop
- [ ] Add coverage for whatever legacy behavior is supported or a clear unsupported-feature error if it is intentionally not supported
## References
- Upstream cleanup: ClickHouse/dbt-clickhouse#644
- MV v2 implementation: ClickHouse/dbt-clickhouse#582
- Macro package: dbt-labs/fs#9883
- Beta materializations (Distributed, Dictionary, S3): dbt-labs/dbt-core#14580
- Parent EPIC: dbt-labs/dbt-fusion#1679
Contributor guide
Assessment
This issue has not been assessed yet.