[EPIC] first class support for struct field / Variant access in Parquet
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
This issue tracks the hard problems that block efficient querying of struct field columns in Parquet
To investigate, I used an MRE based on a variant column represented as a nested struct. Though, the findings generalize to any schema where queries access leaf fields inside struct columns
- https://github.com/apache/datafusion/issues/21306
- https://github.com/apache/datafusion/issues/21307
- https://github.com/apache/datafusion/pull/21003
Contributor guide
Research direction
Start by reviewing issues 21306 and 21307 and pull request 21003, which contain the reported blockers and investigation context. The epic is complete when efficient querying of leaf fields inside nested struct or variant columns in Parquet has first-class support.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100