[Parquet] How to decode part columns of ColumnIndex
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Which part is this question about**
New parquet metadata decoder and page index. I know we have https://github.com/apache/arrow-rs/issues/8643 , and we can avoid Statistics via https://github.com/apache/arrow-rs/blob/main/parquet/src/file/metadata/options.rs#L91
but column index seems follow a different rule: https://github.com/apache/arrow-rs/blob/main/parquet/src/file/metadata/parser.rs#L244
I may miss the reason here.
**Describe your question**
How can we avoid decode some columns in column index
**Additional context**
No
Contributor guide
Research direction
Start by reading the referenced code in parquet/src/file/metadata/options.rs and parquet/src/file/metadata/parser.rs, then review issue 8643 for the related metadata-decoding context. Compare how Statistics and ColumnIndex are decoded and establish the intended rule for skipping columns; done means the expected behavior is clarified well enough to guide an implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100