Support using simdutf8 for `validate_string_view` and other utf8 validation
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We have some different sources were `std::str::from_utf8` is used for utf8 validation, such as `validate_string_view`.
We can speed it up by supporting `simdutf8` as well here.
**Describe the solution you'd like**
Similar for https://github.com/apache/arrow-rs/pull/6668 , optionally switch to using `simdutf8` instead.
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start at the `validate_string_view` entry point and inspect the other uses of `std::str::from_utf8` for UTF-8 validation. Read the approach in apache/arrow-rs#6668 before deciding where `simdutf8` can be used. Done means the relevant validation paths support the faster validator while preserving existing validation behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100