apache / apache/arrow-rs

[Parquet] How to decode part columns of ColumnIndex

Open
#9,609 3 comments 0 reactions 0 assignees View on GitHub
parquet question
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Which part is this question about**

New parquet metadata decoder and page index. I know we have https://github.com/apache/arrow-rs/issues/8643 , and we can avoid Statistics via https://github.com/apache/arrow-rs/blob/main/parquet/src/file/metadata/options.rs#L91
but column index seems follow a different rule: https://github.com/apache/arrow-rs/blob/main/parquet/src/file/metadata/parser.rs#L244

I may miss the reason here.

**Describe your question**

How can we avoid decode some columns in column index

**Additional context**

No

Contributor guide

Open the contributing guide

Research direction

Start by reading the referenced code in parquet/src/file/metadata/options.rs and parquet/src/file/metadata/parser.rs, then review issue 8643 for the related metadata-decoding context. Compare how Statistics and ColumnIndex are decoded and establish the intended rule for skipping columns; done means the expected behavior is clarified well enough to guide an implementation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.