ClickHouse / ClickHouse/dbt-clickhouse
Support for a JDBC source
- Dominant language
- Python
- Stars
- 362
- Forks
- 176
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
When defining a DBT model we can use the source macro to reference an existing table in Clickhouse using database and table name, but in Clickhouse a NamedDataSource can be defined an a table can be referenced using not only `.` but also `jdbc('', '')`.
As it is now in the dbt-clickhouse adapter, there is no way (afaik) to keep the concept that this jdbc bridge reference is a source and access the data, because there is now source macro call we can write that would make it work.
There is a number of solution we could imagine and we'd need to talk about what would be best, my main option would be to :
* use the loader tag in the sources yaml definition, to change the source resolution in the case where loader=jdbc (it is accessible in the SourceDefinition object even if documentation states that it's only for doc purposes, I guess we could use it) and override the ClickHouseRelation.render method.
but we could also
* define an additional macro like jdbc_source (postgresql_source, mysql_source as well I guess) that would proxy the source resolution
I don't think this is doable apart from within the adapter, as it would need to override the rendering process of the relation, and I do think that abandoning the concept of source for jdbc/postgresql/mysql... engines would be a bit sad.
Contributor guide
Research direction
Start by tracing the dbt source macro and the adapter's ClickHouseRelation.render path, then inspect how SourceDefinition.loader is exposed for sources defined in YAML. The issue needs a decided design before implementation; done should mean a JDBC-backed source can resolve its named data source and table or query while retaining source semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100