Fast scalar path for array_slice
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
Current array_slice implementation assumes the from/to/stride arguments come from arrays which may vary per row; however I'd assume most cases we actually have scalar values for those arguments in which case we can take a fast path.
### Describe the solution you'd like
For example in here:
https://github.com/apache/datafusion/blob/9238779f45418e07108079aeef3b51de85e9fb8f/datafusion/functions-nested/src/extract.rs#L456-L470
If we know from the scalar from/to/stride that the slice is either empty or contiguous, we can avoid recreating the child array as no shuffling is needed; only in cases where the stride is > 1 or if its a reverse slice would we need to actually shuffle child array data.
### Describe alternatives you've considered
_No response_
### Additional context
Probably makes more sense to do this after the refactoring done by #18432
Contributor guide
Research direction
Start with datafusion/functions-nested/src/extract.rs around lines 456-470 and review the refactoring planned in issue #18432. Determine how scalar from/to/stride values identify empty or contiguous slices, and verify that the fast path preserves behavior while avoiding child-array shuffling except for reverse slices or strides greater than one.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100