dlt-hub / dlt-hub/dlt

[feat] allow custom merge filter expression for delta lake

Open
#2,279 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.9k
Forks
605
Avg merge
1d 14h
Merged PRs (30d)
38

Description

**Background**
`delta` merge job can be easily customized to execute user-provided filter expression.

**Requirements**
we can let users pass any update expression to the `merge_delta_table` function:

* if you have a `dlt.resource` you can use `apply_hints` with `additional_table_hints` to add any table level hint
* make the `merge_delta_table` function to look for ie. `x-merge-update-filter` in schema that contains the expression you want to add to the `update_all`
* if found just pass it to the `update_all`.
this will allow for any filter expression user may want to be passed.

**Implementation**
We can create `delta_adapter` resource adapter that will add custom expression to a resource (and do other things ie. will allow to define partitions on columns)

Contributor guide

Open the contributing guide

Research direction

Start by locating the merge_delta_table function and the Delta adapter or resource-adapter entry points, then read how dlt.resource, apply_hints, additional_table_hints, and update_all are currently handled. Done means a schema value such as x-merge-update-filter can provide an expression that reaches update_all, with the proposed delta_adapter behavior clarified and covered by the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.