datafusion-contrib / datafusion-contrib/liquid-cache
[Epic] Support Variant type
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 452
- Forks
- 51
- Avg merge
- 3h 10m
- Merged PRs (30d)
- 12
Description
Parquet has recently added variant type. It's a super important data type that LiquidCache needs to handle properly.
https://github.com/apache/arrow-rs/issues/8480
- Add basic Variant type support
- Improve shredding
I think this paper is particularly relevant: https://db.in.tum.de/~durner/papers/json-tiles-sigmod21.pdf
The key idea behind JSON tiles is to automatically detect the im-
plicit common structure across a collection of objects. Using this
structural information, we infer types, materialize frequently oc-
curring keys as relational columns, and collect query optimizer
statistics – enabling performance close to that of native relational
column stores. Infrequent keys and heterogeneous (outlier) objects
are stored in an optimized binary format that allows fast access to
individual keys.
This is what we want to do. But we want to do this in the caching context.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named. Start by reviewing the linked Apache Arrow issue and the JSON Tiles paper, then examine how LiquidCache currently handles Parquet data. Done means adding basic Variant type support and improving shredding for the caching context described here.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100