apache / apache/arrow-rs

[Parquet] Virtual Column Support

Open
#8,799 0 comments 0 reactions 0 assignees View on GitHub
enhancement parquet
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
When reading data from Parquet files sometimes you want to know the lineage of the rows that come out so you can identify them in the future.

For example, when implementing delete predicates, you may want to find all rows that match a particular filter condition and remember their row numbers in the file (e.g. how iceberg works)

Or you may want to build up a secondary index with information on the min/max values for each row group or data page

Sometimes this information can be determined by reading all the rows from the parquet file and reconstructing the row number (or row group number), but this is slow if predicates are applied during the scan, for example, in which case the row numbers are needed from the reader itself.

Information that is hard to

**Describe the solution you'd like**
Add "virtual" column support to the Parquet reader

**Additional context**
- [x] https://github.com/apache/arrow-rs/issues/7299
- [x] https://github.com/apache/arrow-rs/issues/8641

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.