apache / apache/arrow-rs

Support using simdutf8 for `validate_string_view` and other utf8 validation

Open
#7,014 7 comments 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

We have some different sources were `std::str::from_utf8` is used for utf8 validation, such as `validate_string_view`.
We can speed it up by supporting `simdutf8` as well here.

**Describe the solution you'd like**

Similar for https://github.com/apache/arrow-rs/pull/6668 , optionally switch to using `simdutf8` instead.

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start at the `validate_string_view` entry point and inspect the other uses of `std::str::from_utf8` for UTF-8 validation. Read the approach in apache/arrow-rs#6668 before deciding where `simdutf8` can be used. Done means the relevant validation paths support the faster validator while preserving existing validation behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.