apache / apache/datafusion

Fast scalar path for array_slice

Open
#18,458 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Current array_slice implementation assumes the from/to/stride arguments come from arrays which may vary per row; however I'd assume most cases we actually have scalar values for those arguments in which case we can take a fast path.

### Describe the solution you'd like

For example in here:

https://github.com/apache/datafusion/blob/9238779f45418e07108079aeef3b51de85e9fb8f/datafusion/functions-nested/src/extract.rs#L456-L470

If we know from the scalar from/to/stride that the slice is either empty or contiguous, we can avoid recreating the child array as no shuffling is needed; only in cases where the stride is > 1 or if its a reverse slice would we need to actually shuffle child array data.

### Describe alternatives you've considered

_No response_

### Additional context

Probably makes more sense to do this after the refactoring done by #18432

Contributor guide

Open the contributing guide

Research direction

Start with datafusion/functions-nested/src/extract.rs around lines 456-470 and review the refactoring planned in issue #18432. Determine how scalar from/to/stride values identify empty or contiguous slices, and verify that the fast path preserves behavior while avoiding child-array shuffling except for reverse slices or strides greater than one.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.