Filtering optimizations
Open
- Dominant language
- Rust
- Stars
- 1.6k
- Forks
- 145
- PR merge metrics
- No merged PRs in 30d
Description
Currently filtering records is always executed after a product. The reason for this is because the filter condition might be applied using a function requiring inputs from multiple joined tables. However this is is not always the case.
Thus, we should split the filtering into two phases:
- First phase is executed at the sources, filtering data based on original fields conditions. These filters should have the option to support filter pushdowns to the original source during the snapshotting phase.
- Second phase is only reserved for filters that require multiple joined tables. These filters shall be applied after a product
Contributor guide
Assessment
This issue has not been assessed yet.