getdozer / getdozer/dozer

Filtering optimizations

Open
#1,967 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.6k
Forks
145
PR merge metrics
No merged PRs in 30d

Description

Currently filtering records is always executed after a product. The reason for this is because the filter condition might be applied using a function requiring inputs from multiple joined tables. However this is is not always the case.

Thus, we should split the filtering into two phases:

- First phase is executed at the sources, filtering data based on original fields conditions. These filters should have the option to support filter pushdowns to the original source during the snapshotting phase.

- Second phase is only reserved for filters that require multiple joined tables. These filters shall be applied after a product

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.