apache / apache/datafusion

[EPIC] first class support for struct field / Variant access in Parquet

Open
#21,308 2 comments 6 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

This issue tracks the hard problems that block efficient querying of struct field columns in Parquet

To investigate, I used an MRE based on a variant column represented as a nested struct. Though, the findings generalize to any schema where queries access leaf fields inside struct columns

- https://github.com/apache/datafusion/issues/21306
- https://github.com/apache/datafusion/issues/21307
- https://github.com/apache/datafusion/pull/21003

Contributor guide

Open the contributing guide

Research direction

Start by reviewing issues 21306 and 21307 and pull request 21003, which contain the reported blockers and investigation context. The epic is complete when efficient querying of leaf fields inside nested struct or variant columns in Parquet has first-class support.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.