datafusion-contrib / datafusion-contrib/liquid-cache
Pushdown predicates that contains `variant_get`
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 452
- Forks
- 51
- Avg merge
- 3h 10m
- Merged PRs (30d)
- 12
Description
Let's consider this query:
SELECT * FROM table
WHERE variant_get(d, "location.country", UTF8) = "US";
My understanding is that variant_get as a UDF will not be pushdown to data provider.
So DataFusion will read the entire variant (potentially shredded) and then perform filter.
This is suboptimal for many reasons, and we should pushdown this to liquid cache.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names the variant_get UDF, the data provider, and the liquid cache, but no files or tests. Start by tracing how the query filter reaches the data provider and where liquid-cache pushdown is handled. Done means predicates containing variant_get can be pushed down without reading the entire variant.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100