hsivonen / hsivonen/encoding_rs
Port UTF-8 to UTF-16 and UTF-16 to UTF-8 conversions from simdutf
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 464
- Forks
- 70
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 2
Description
Figure out the portable commonality of the AVX2 and aarch64 implementations in https://github.com/simdutf/simdutf/ and port them to core::simd and possible some vendor operations.
This should probably go into a separate crate to allow use without the non-UTF encodings supported by encoding_rs.
Instead of first validating the whole input, perhaps piecewise UTF-8 validation from the simdutf8 crate either needs to be exposed to the UTF-8 to UTF-16 conversion on copypasted.
Since the UTF-16 conversions use very large lookup tables, application-context benchmarking is probably needed to understand the cache effects and to decide what input is long enough for the extra cache evictions to be worth it.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the AVX2 and aarch64 implementations in simdutf, then inspect core::simd, vendor operations, and the simdutf8 crate for portable UTF-8 validation. Define a separate-crate approach for UTF-8/UTF-16 conversion and benchmark realistic input sizes to evaluate lookup-table cache effects and determine when the optimized path is worthwhile.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100