apache / apache/arrow

[C++][Parquet] Skip method skips levels and not rows for repeated fields

Open
#42,995 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Component: Parquet Priority: Major Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

The implementation of TypedColumnReader::Skip method with signature:

virtual int64_t Skip(int64_t num_levels_to_skip) = 0;

will skip levels for both repeated fields and non-repeated fields. We want to be able to skip rows for repeated fields, and skipping levels is not that useful.

For example, for the following rows:

message M \{ repeated int32 b = 1 }

rows: {}, \{[10,10]}, \{[20, 20, 20]}

values = \{10, 10, 20, 20, 20};
def_levels = \{0, 1, 1, 1, 1, 1};
rep_levels = \{0, 0, 1, 0, 1, 1};

We want skip(2) to skip the first two rows, so that the next value that we read is 20. However, it will skip the first two levels, and the next value that we read is 10.

**Reporter**: [fatemah](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=panahi) / @fatemehp

**Note**: *This issue was originally created as [PARQUET-2175](https://issues.apache.org/jira/browse/PARQUET-2175). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

Open the contributing guide

Research direction

Locate the implementation of TypedColumnReader::Skip and any tests covering repeated-field definition and repetition levels. Reproduce the example with rows {}, {[10,10]}, {[20,20,20]} and verify that Skip(2) advances past the first two rows so the next value read is 20.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.