datafusion-contrib / datafusion-contrib/liquid-cache

[Epic] Support Variant type

Open
#392 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
452
Forks
51
Avg merge
3h 10m
Merged PRs (30d)
12

Description

Parquet has recently added variant type. It's a super important data type that LiquidCache needs to handle properly.

https://github.com/apache/arrow-rs/issues/8480

  • Add basic Variant type support
  • Improve shredding

I think this paper is particularly relevant: https://db.in.tum.de/~durner/papers/json-tiles-sigmod21.pdf

The key idea behind JSON tiles is to automatically detect the im-
plicit common structure across a collection of objects. Using this
structural information, we infer types, materialize frequently oc-
curring keys as relational columns, and collect query optimizer
statistics – enabling performance close to that of native relational
column stores. Infrequent keys and heterogeneous (outlier) objects
are stored in an optimized binary format that allows fast access to
individual keys.

This is what we want to do. But we want to do this in the caching context.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by reviewing the linked Apache Arrow issue and the JSON Tiles paper, then examine how LiquidCache currently handles Parquet data. Done means adding basic Variant type support and improving shredding for the caching context described here.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.