ClickHouse / ClickHouse/dbt-clickhouse

Support for External Tables in dbt-clickhouse

Open
#585 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

## Description

External tables in ClickHouse (table engines like Kafka, Postgres, S3) are technically infra objects that handle credentials and have different lifecycles (compared to regular data modeling objects). This creates a conceptual difference with dbt's core design principles, which are more focused on data transformation rather than data ingestion or infrastructure management.

## Current Situation

We currently have partial support for some table engines (S3, Memory) through the `IGNORED_SETTINGS` mechanism that excludes MergeTree-specific settings like `replicated_deduplication_window`. However, this solution is limited and doesn't account for the broader questions around:

1. Whether external tables belong in the adapter at all
2. How to handle credentials securely
3. What should be the proper abstraction

## Questions to address

### 1. Implementation Approach

We need to decide between:

- **Native adapter support** - Extend the adapter to fully support table engines as a materialization type
- **External tables package** - Use dbt-labs' `dbt-external-tables` [package](https://github.com/dbt-labs/dbt-external-tables)
- **Hybrid approach** - Document recommendations for managing external tables outside dbt (e.g., via Terraform?)

### 2. Credential Management

External tables require credentials, which creates security issues:

- **Named Collections**: Named collections are supported on the OSS version, and will be available in the ClickHouse cloud as well. It would allow referencing credentials without hardcoding them in dbt config files
- **Environment variables**: Current workarounds require managing credentials through environment variables
- **Terraform integration**: Some users prefer managing infrastructure tables separately using IaC tools

### 3. Alignment with dbt Core

See: https://github.com/dbt-labs/dbt-core/discussions/11265

## Next Steps (TBD)

1. Research how other adapters (Snowflake, BigQuery, Databricks) handle external tables
2. Evaluate the `dbt-external-tables` package with ClickHouse
3. Document recommended patterns for external table management (potentially using Terraform + dbt)
4. Once Named Collections are available in ClickHouse Cloud, re-evaluate security concerns

## Related Issues

- #500 - Kafka table engine support
- #296
- #372, #366 - Settings compatibility with non-MergeTree engines

---

We would love to get the community input on how it is being done today, what requirements you might need, and how overall your usage is with external tables with ClickHouse

Contributor guide

Open the contributing guide

Research direction

Start by researching how the Snowflake, BigQuery, and Databricks adapters handle external tables, then evaluate the dbt-external-tables package for ClickHouse. Compare native, package-based, and Terraform-managed approaches, including Named Collections and environment-variable credentials; done means a community-backed recommendation and documented management pattern.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.