datafusion-contrib / datafusion-contrib/liquid-cache
Support pushing down row selection
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 452
- Forks
- 51
- Avg merge
- 3h 10m
- Merged PRs (30d)
- 12
Description
I had a nice discussion with @mbutrovich today about pushing down row selection to the storage service.
Specifically, consider that we want to join two tables, A is super small, and B is super large. A is essentially a filter. The goal is only to load B's rows that match rows in A.
This will require us to pass a row selection down to the storage service. I think DataFusion doesn't support sideways information passing yet: https://github.com/apache/datafusion/issues/7955
Once DataFusion implements that, we also need to make Parquet support it: https://github.com/apache/datafusion/issues/14816
I think DuckDB implements a simple version of this: https://github.com/duckdb/duckdb/pull/12908
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the linked DataFusion issues on sideways information passing and Parquet support, then compare DuckDB's linked implementation. Define how row selections would be passed to the storage service and what matching rows from the large table should be loaded; the work is complete when this design is implemented and validated for the stated join scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100