apache / apache/arrow-rs

Consider removing `skip` from `RowSelector`

Open
#7,450 9 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

Currently `RowSelector` contains a `pub skip: bool` which means that the rows in the row selector needs to be skipped or not.
However, this field is not very useful as an optimal representation of a `RowSelector` will always be alternating selected and skipped rows.

I think we should consider dropping the `skip` field in order to simplify the api (and speed up / reduce memory overhead (from 16 to 8 bytes per element) as well).

We can represent a `RowSelector` as array of alternating select / skip / select rows.

e.g. :
[0, 10, 5, 10, 5] => select 0, skip 10, select 5, skip 10, select 5

**Describe the solution you'd like**

Drop the `skip` field, update implementation to take care of the new representation (select / skip based on alternation rather than via the field, (even/odd)).

**Describe alternatives you've considered**

Other representation, e.g. Vec>

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Locate the RowSelector definition and its constructors and consumers; inspect how the current skip field determines selection. Update the representation and dependent implementation to use alternating select/skip positions, then verify existing RowSelector behavior and the stated memory representation.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.