Add with_virtual_columns to ParquetSource for reading virtual columns
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
I have found that arrow-rs has supported reading virtual columns, including RowGroupIndex and RowNumber, since v57.0.0.
I propose to add a method `with_virtual_columns` to `ParquetSource` to let user decide which virtual columns should be read.
The virtual columns, like RowNumber, could be used to implement prewhere, late materialized topk, and secondary indexes and more fancy stuff.
virtual columns support in arrow-rs: https://github.com/apache/arrow-rs/pull/8715
Contributor guide
Research direction
Start by locating ParquetSource and reviewing the arrow-rs virtual-column support introduced in v57.0.0, including RowGroupIndex and RowNumber. Trace how ParquetSource selects columns and identify the relevant tests; done means users can choose virtual columns through with_virtual_columns and the behavior is covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100