apache / apache/arrow-rs

Optimize infallible dictionary key casting in `try_new_from_builder` method

Open
#7,662 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

**Describe the solution you'd like**

In https://github.com/apache/arrow-rs/pull/7611 we added a new method to the dictionary builders called `try_new_from_builder` that allows creating a new builder with the same values, but a different type for the dictionary keys.

This method uses the `PrimitiveArray::try_unary` method to do the casting, but when casting infallible types (e.g. going from u8 -> u16 keys), it would probably be faster to use `PrimitiveArray::unary_mut`.

We should improve the implementation of `try_new_from_builder` to use `unary_mut` where possible. It might also be good to have add benchmark for this method as part of this work, so we could measure if there really is a performance improvement.

**Describe alternatives you've considered**

**Additional context**

Discussion https://github.com/apache/arrow-rs/pull/7611/files#r2143611512

Contributor guide

Open the contributing guide

Research direction

Start by reading the `try_new_from_builder` implementation and the `PrimitiveArray::try_unary` and `unary_mut` entry points, then review PR 7611 for the builder context. Update the casting path for infallible key conversions and add a benchmark if the existing benchmark structure supports it; done means the implementation is covered by tests and the performance can be compared.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.