Parquet: How to do concurrent decoding over columns?
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Which part is this question about**
parquet's api.
**Describe your question**
How to do concurrent read/decoding over columns? The similar code can be found in [polars](https://github.com/pola-rs/polars/blob/ab83c2ad27b82148617af1f00bb26ae809c4aa04/crates/polars-io/src/parquet/read_impl.rs#L166)
I found the `build_array_reader` function, but I can't construct a `ParquetField`, which is the input param of this function.
**Additional context**
Contributor guide
Research direction
Start with the `build_array_reader` function and the `ParquetField` input mentioned in the issue, then compare the linked Polars `read_impl.rs` implementation. Determine whether the existing Parquet API exposes a supported way to construct the field and perform concurrent column decoding; done means documenting the applicable entry points and usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100