apache / apache/arrow-rs

Value type for TypedRunArray<I, V>

Open
#9,458 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

We cannot fully type (on index & value types) a run-end-encoded array value.

| What | Value | Reference |
|-|-|-|
| Array | `dyn Array` | `&dyn Array` |
| RunArray | `RunArray` | `&RunArray` |
| TypedRunArray | zilch! | `TypedRunArray<'a, I, V>` |

We can have a value describing a RunArray with a known index type. And we can take a reference to it.

But for a RunArray with known index and value types, we can only have a reference type, not a value type. This means that we cannot have a function return a fresh, fully described run array, with a known value type. We can only return a partially-typed RunArray, and add a comment saying what the values are. For example, consider the fictional:

```
/// Returns a RunArray whose values are `A`.
fn encode_array(array: A) -> RunArray {
todo!()
}
```
It would be nice to return a `TypedRunArray`. But that's not possible. We can only add a comment, and let the user cast as needed.

**Describe the solution you'd like**

I'd rather see TypedRunArray a value type.

An issue is that an Array cannot both be a `struct RunArray` and a `struct TypedRunArray` so there's some type design to do.

Additionally, it would be a backwards-imcompatible change. TypedRunArray might have to stay as-is. Maybe introduce `TypedRunArrayValue`, and one can make a `TypedRunArray<'_>` from either a `&RunArray` or a `&TypedRunArrayValue`.

**Additional context**

Another defect of the current `TypedRunArray` is that `V` is not constrained to `impl Array`. Not sure that's intended.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing TypedRunArray and RunArray types and their relationships. The issue identifies value semantics, compatibility, and whether to introduce TypedRunArrayValue as the open design questions; done means a fully typed run-end-encoded array value can be returned while the backward-incompatible impact is resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.