ClickHouse / ClickHouse/dbt-clickhouse
Support update_field on dictionary materializations
- Dominant language
- Python
- Stars
- 362
- Forks
- 177
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 8
Description
**Is your feature request related to a problem? Please describe.**
Dictionaries that are backed by large clickhouse tables/queries typically are reloaded from the entire dataset. This may be inefficient if only a few rows have changed. There is an option to specify a field that enables new data to be identified, thereby minimizing the updated data.
**Describe the solution you'd like**
Support the `update_field` syntax in the dictionary materialization
**Additional context**
[Add any other context or screenshots about the feature request here.](https://clickhouse.com/docs/en/sql-reference/dictionaries#refreshing-dictionary-data-using-lifetime)
```
SOURCE(CLICKHOUSE(...
update_field 'added_time'
QUERY '
SELECT my_arr.1 AS x, my_arr.2 AS y, creation_time
FROM (
SELECT arrayZip(x_arr, y_arr) AS my_arr, creation_time
FROM dictionary_source
WHERE {condition}
)'
))
```
Contributor guide
Research direction
Start with the linked ClickHouse documentation on refreshing dictionary data and trace how dictionary materializations are handled in this dbt-clickhouse repository. Identify the materialization entry point and any related tests, then verify that the shown update_field syntax is accepted and produces incremental dictionary refresh behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100