apache / apache/arrow-rs

Parquet: How to do concurrent decoding over columns?

Open
#5,120 4 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Which part is this question about**

parquet's api.

**Describe your question**

How to do concurrent read/decoding over columns? The similar code can be found in [polars](https://github.com/pola-rs/polars/blob/ab83c2ad27b82148617af1f00bb26ae809c4aa04/crates/polars-io/src/parquet/read_impl.rs#L166)

I found the `build_array_reader` function, but I can't construct a `ParquetField`, which is the input param of this function.

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with the `build_array_reader` function and the `ParquetField` input mentioned in the issue, then compare the linked Polars `read_impl.rs` implementation. Determine whether the existing Parquet API exposes a supported way to construct the field and perform concurrent column decoding; done means documenting the applicable entry points and usage.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.