apache / apache/arrow

[C++][Parquet] Add Skip on Parquet decoders

Open
#50,453 0 comments 1 reaction 0 assignees View on GitHub
Component: C++ Component: Parquet Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Add `Skip` on `TypedDecoder` in Parquet encoding.

https://github.com/apache/arrow/blob/3b515ede5fa270bab8ebbaba9afef816854a3708/cpp/src/parquet/encoding.h#L267

The absence of this function means that downstream users have to allocate and and decode into a scratch buffer to be able to throw away values.
While not all decoder can have an efficient `Skip` implementation, some can, and the remaining can adapt the value of a stack buffer to a size appropriate for their own `Decode` function.

The decoding in a scratch buffer is currently being privately implemented in a temporary wrapper around the decoder in
https://github.com/apache/arrow/blob/cc90e40e5703f13ef1e38dd728eda93961d336d6/cpp/src/parquet/column_reader.cc#L672

### Component(s)

C++, Parquet

Contributor guide

Open the contributing guide

Research direction

Start in cpp/src/parquet/encoding.h at TypedDecoder::Skip and compare the decoder implementations that would need the API. Read the temporary scratch-buffer wrapper in cpp/src/parquet/column_reader.cc around line 672 to understand the current behavior. Done means decoders can discard values through the new operation without requiring downstream callers to allocate a decode buffer.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.