[feat] allow custom merge filter expression for delta lake
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 605
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
**Background**
`delta` merge job can be easily customized to execute user-provided filter expression.
**Requirements**
we can let users pass any update expression to the `merge_delta_table` function:
* if you have a `dlt.resource` you can use `apply_hints` with `additional_table_hints` to add any table level hint
* make the `merge_delta_table` function to look for ie. `x-merge-update-filter` in schema that contains the expression you want to add to the `update_all`
* if found just pass it to the `update_all`.
this will allow for any filter expression user may want to be passed.
**Implementation**
We can create `delta_adapter` resource adapter that will add custom expression to a resource (and do other things ie. will allow to define partitions on columns)
Contributor guide
Research direction
Start by locating the merge_delta_table function and the Delta adapter or resource-adapter entry points, then read how dlt.resource, apply_hints, additional_table_hints, and update_all are currently handled. Done means a schema value such as x-merge-update-filter can provide an expression that reaches update_all, with the proposed delta_adapter behavior clarified and covered by the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100