ClickHouse / ClickHouse/dbt-clickhouse
Support for External Tables in dbt-clickhouse
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
## Description
External tables in ClickHouse (table engines like Kafka, Postgres, S3) are technically infra objects that handle credentials and have different lifecycles (compared to regular data modeling objects). This creates a conceptual difference with dbt's core design principles, which are more focused on data transformation rather than data ingestion or infrastructure management.
## Current Situation
We currently have partial support for some table engines (S3, Memory) through the `IGNORED_SETTINGS` mechanism that excludes MergeTree-specific settings like `replicated_deduplication_window`. However, this solution is limited and doesn't account for the broader questions around:
1. Whether external tables belong in the adapter at all
2. How to handle credentials securely
3. What should be the proper abstraction
## Questions to address
### 1. Implementation Approach
We need to decide between:
- **Native adapter support** - Extend the adapter to fully support table engines as a materialization type
- **External tables package** - Use dbt-labs' `dbt-external-tables` [package](https://github.com/dbt-labs/dbt-external-tables)
- **Hybrid approach** - Document recommendations for managing external tables outside dbt (e.g., via Terraform?)
### 2. Credential Management
External tables require credentials, which creates security issues:
- **Named Collections**: Named collections are supported on the OSS version, and will be available in the ClickHouse cloud as well. It would allow referencing credentials without hardcoding them in dbt config files
- **Environment variables**: Current workarounds require managing credentials through environment variables
- **Terraform integration**: Some users prefer managing infrastructure tables separately using IaC tools
### 3. Alignment with dbt Core
See: https://github.com/dbt-labs/dbt-core/discussions/11265
## Next Steps (TBD)
1. Research how other adapters (Snowflake, BigQuery, Databricks) handle external tables
2. Evaluate the `dbt-external-tables` package with ClickHouse
3. Document recommended patterns for external table management (potentially using Terraform + dbt)
4. Once Named Collections are available in ClickHouse Cloud, re-evaluate security concerns
## Related Issues
- #500 - Kafka table engine support
- #296
- #372, #366 - Settings compatibility with non-MergeTree engines
---
We would love to get the community input on how it is being done today, what requirements you might need, and how overall your usage is with external tables with ClickHouse
Contributor guide
Research direction
Start by researching how the Snowflake, BigQuery, and Databricks adapters handle external tables, then evaluate the dbt-external-tables package for ClickHouse. Compare native, package-based, and Terraform-managed approaches, including Named Collections and environment-variable credentials; done means a community-backed recommendation and documented management pattern.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100