ClickHouse / ClickHouse/dbt-clickhouse

Support update_field on dictionary materializations

Open
#352 0 comments 0 reactions 0 assignees View on GitHub
enhancement good first issue
Dominant language
Python
Stars
362
Forks
177
Avg merge
2d 10h
Merged PRs (30d)
8

Description

**Is your feature request related to a problem? Please describe.**
Dictionaries that are backed by large clickhouse tables/queries typically are reloaded from the entire dataset. This may be inefficient if only a few rows have changed. There is an option to specify a field that enables new data to be identified, thereby minimizing the updated data.

**Describe the solution you'd like**
Support the `update_field` syntax in the dictionary materialization

**Additional context**
[Add any other context or screenshots about the feature request here.](https://clickhouse.com/docs/en/sql-reference/dictionaries#refreshing-dictionary-data-using-lifetime)

```
SOURCE(CLICKHOUSE(...
update_field 'added_time'
QUERY '
SELECT my_arr.1 AS x, my_arr.2 AS y, creation_time
FROM (
SELECT arrayZip(x_arr, y_arr) AS my_arr, creation_time
FROM dictionary_source
WHERE {condition}
)'
))
```

Contributor guide

Open the contributing guide

Research direction

Start with the linked ClickHouse documentation on refreshing dictionary data and trace how dictionary materializations are handled in this dbt-clickhouse repository. Identify the materialization entry point and any related tests, then verify that the shown update_field syntax is accepted and produces incremental dictionary refresh behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, python
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.