ClickHouse / ClickHouse/dbt-clickhouse

Support ClickHouse named collections

Open
#658 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

## Is your feature request related to a problem?

ClickHouse [named collections](https://clickhouse.com/docs/operations/named-collections) let you store reusable key-value config (credentials, URLs, connection settings) for integrations like S3, RabbitMQ, Kafka, and remote ClickHouse. dbt-clickhouse currently has no way to create or manage them, so users must define them manually outside of dbt or embed raw `CREATE NAMED COLLECTION` statements in `on-run-start`/hooks, which sit outside dbt's dependency graph and state.

## Describe the solution you'd like

Named collections should be defined like any other dbt resource — via `{{ config(...) }}` or a `.yml` file — and created/updated as part of a `dbt run`.

For example, a new materialization driven by config:

```sql
-- models/collections/s3_mydata.sql
{{ config(
materialized='named_collection',
keys={
'access_key_id': {'value': env_var('S3_KEY'), 'overridable': false},
'secret_access_key': {'value': env_var('S3_SECRET'), 'overridable': false},
'url': {'value': 'https://bucket.s3.amazonaws.com/', 'overridable': true}
}
) }}
```

or equivalently in YAML:

```yaml
named_collections:
- name: s3_mydata
keys:
access_key_id: { value: "{{ env_var('S3_KEY') }}", overridable: false }
secret_access_key: { value: "{{ env_var('S3_SECRET') }}", overridable: false }
url: { value: "https://bucket.s3.amazonaws.com/", overridable: true }
```

Models and sources could then reference the collection by name:

```sql
SELECT * FROM s3({{ ref('s3_mydata') }}, filename = 'test.tsv')
```
I am not sure whether `ref` is the right macro to use here, as it usually refers to a table, not an identifier.

dbt should reconcile state on each run — `CREATE`, `ALTER ... SET ... DELETE` to apply changes, and respect `OVERRIDABLE` / `NOT OVERRIDABLE` and `ON CLUSTER` for clustered deployments.

### Additional context

Docs: https://clickhouse.com/docs/operations/named-collections

Contributor guide

Open the contributing guide

Research direction

No repository files or tests are named. Start by reviewing existing dbt materializations and how resource configuration is parsed, then compare those patterns with ClickHouse named collection syntax and state reconciliation. Done should include a settled configuration/reference design plus create, update, deletion, overridability, and ON CLUSTER behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.