datafusion-contrib / datafusion-contrib/liquid-cache
[Epic] Struct shredding support
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 452
- Forks
- 51
- Avg merge
- 3h 10m
- Merged PRs (30d)
- 12
Description
Very similar (but much simpler) to variant #426
The idea is simple, let's say we have a struct array:
struct Location {
country: String
zipcode: Int32
}
And in the sql, we only read the zipcode, SELECT location.zipcode from table, then it makes no sense to cache the entire Location column.
To be fair, the better approach is for DataFusion to only read zipcode column from Parquet. I believe this is been tracked by https://github.com/apache/datafusion/issues/2581
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the related variant issue #426 and the DataFusion tracking issue #2581 to understand the intended boundaries. Define how projected fields such as location.zipcode should be identified and cached without caching the full struct, then verify the behavior with representative struct-array queries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100