ClickHouse / ClickHouse/dbt-clickhouse
Support ClickHouse named collections
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
## Is your feature request related to a problem?
ClickHouse [named collections](https://clickhouse.com/docs/operations/named-collections) let you store reusable key-value config (credentials, URLs, connection settings) for integrations like S3, RabbitMQ, Kafka, and remote ClickHouse. dbt-clickhouse currently has no way to create or manage them, so users must define them manually outside of dbt or embed raw `CREATE NAMED COLLECTION` statements in `on-run-start`/hooks, which sit outside dbt's dependency graph and state.
## Describe the solution you'd like
Named collections should be defined like any other dbt resource — via `{{ config(...) }}` or a `.yml` file — and created/updated as part of a `dbt run`.
For example, a new materialization driven by config:
```sql
-- models/collections/s3_mydata.sql
{{ config(
materialized='named_collection',
keys={
'access_key_id': {'value': env_var('S3_KEY'), 'overridable': false},
'secret_access_key': {'value': env_var('S3_SECRET'), 'overridable': false},
'url': {'value': 'https://bucket.s3.amazonaws.com/', 'overridable': true}
}
) }}
```
or equivalently in YAML:
```yaml
named_collections:
- name: s3_mydata
keys:
access_key_id: { value: "{{ env_var('S3_KEY') }}", overridable: false }
secret_access_key: { value: "{{ env_var('S3_SECRET') }}", overridable: false }
url: { value: "https://bucket.s3.amazonaws.com/", overridable: true }
```
Models and sources could then reference the collection by name:
```sql
SELECT * FROM s3({{ ref('s3_mydata') }}, filename = 'test.tsv')
```
I am not sure whether `ref` is the right macro to use here, as it usually refers to a table, not an identifier.
dbt should reconcile state on each run — `CREATE`, `ALTER ... SET ... DELETE` to apply changes, and respect `OVERRIDABLE` / `NOT OVERRIDABLE` and `ON CLUSTER` for clustered deployments.
### Additional context
Docs: https://clickhouse.com/docs/operations/named-collections
Contributor guide
Research direction
No repository files or tests are named. Start by reviewing existing dbt materializations and how resource configuration is parsed, then compare those patterns with ClickHouse named collection syntax and state reconciliation. Done should include a settled configuration/reference design plus create, update, deletion, overridability, and ON CLUSTER behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100