apache / apache/arrow-rs

Blog / paper post about parquet push decoder

Open
#8,035 5 comments 0 reactions 1 assignee Claimed by @alamb View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- Part of https://github.com/apache/arrow-rs/issues/8000

- Related to https://github.com/apache/arrow-rs/issues/7983

**Describe the solution you'd like**
I would like to explain the benefits of a push decoder in a more digestable format than a PR (even if the code is well documented :bowtie: )

I think this will also help reviewers

**Describe alternatives you've considered**

Idea is to write a blog post about the push parquet decoder, and how it can be used to offer more fine grained control over IO and CPU work in the parquet reader.

The blog post would cover:
1. Motivation: why do we need a push decoder?
2. Design: how does the push decoder work?
3. Examples: how to use the push decoder in practice
4. Performance: how does the push decoder perform compared to the existing parquet reader?
5. Future work: what are the next steps for the push decoder?

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.