Support UTF8 sequence length
- Dominant language
- Rust
- Stars
- 1.1k
- Forks
- 77
- PR merge metrics
- No merged PRs in 30d
Description
I think it would be good to expose a free function from `bstr` that exposes some decoding-specific information about what a given byte means in the context of utf8.
Accessing this info is low level, but has various use cases -- some examples include finding a place to start parsing from given an index, finding a legal cutoff position if you need to truncate a buffer... Etc. (Let me know if you want more cases, I feel like I run into it a fair bit when working with partially invalid utf8).
Specifically, something like this:
```rust
// If `b` indicates the start of a utf8 sequnence boundary,
// returns `Some(sequence_len)`. Returns `None` for all other cases.
pub fn utf8_sequence_len(b: u8) -> Option;
```
Or... Maybe. I'd kinda like to distinguish between valid-but-not-leading and always-invalid bytes. Returning an enum maybe? Thoughts and bikeshedding welcome, I think in practice this would be useful, but also wanted to keep the things small and simple.
---
That said, I do feel strongly that this should not be methods on byteslice like `ByteSlice::is_char_boundary(&self, index: usize) -> bool` and `ByteSlice::utf8_sequence_len(&self, index: usize) -> Option` (mentioning mostly because I suggested these in #42) -- I think those two would be very confusing in practice:
- `ByteSlice::is_char_boundary` would have to return different results from `str::is_char_boundary` even for a fully utf8 byte slice (example: index == len). Having the caller get the byte in question avoids this issue. (Renaming it doesn't even really solve this problem -- still seems like it could cause confusion if 0/len are not conidered boundaries).
- `ByteSlice::utf8_sequence_len(&self, idx)` could behave too many ways -- specifically IDK if it only reads `self[idx]` or if it considers other bytes nearby (e.g. if it's not a leading byte). Making it a top level function only taking a `u8` removes this ambiguity -- reasonably only one thing it could do
Contributor guide
No contributing guide indexed for this repository
Research direction
Begin by reviewing the bstr public API and existing UTF-8 handling; no file or test is named. Resolve whether the API distinguishes leading, continuation, and always-invalid bytes, then define focused coverage for the selected behavior and return shape.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100