BurntSushi / BurntSushi/bstr

Support UTF8 sequence length

Open
#44 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
1.1k
Forks
77
PR merge metrics
No merged PRs in 30d

Description

I think it would be good to expose a free function from `bstr` that exposes some decoding-specific information about what a given byte means in the context of utf8.

Accessing this info is low level, but has various use cases -- some examples include finding a place to start parsing from given an index, finding a legal cutoff position if you need to truncate a buffer... Etc. (Let me know if you want more cases, I feel like I run into it a fair bit when working with partially invalid utf8).

Specifically, something like this:

```rust
// If `b` indicates the start of a utf8 sequnence boundary,
// returns `Some(sequence_len)`. Returns `None` for all other cases.
pub fn utf8_sequence_len(b: u8) -> Option;
```

Or... Maybe. I'd kinda like to distinguish between valid-but-not-leading and always-invalid bytes. Returning an enum maybe? Thoughts and bikeshedding welcome, I think in practice this would be useful, but also wanted to keep the things small and simple.

---

That said, I do feel strongly that this should not be methods on byteslice like `ByteSlice::is_char_boundary(&self, index: usize) -> bool` and `ByteSlice::utf8_sequence_len(&self, index: usize) -> Option` (mentioning mostly because I suggested these in #42) -- I think those two would be very confusing in practice:

- `ByteSlice::is_char_boundary` would have to return different results from `str::is_char_boundary` even for a fully utf8 byte slice (example: index == len). Having the caller get the byte in question avoids this issue. (Renaming it doesn't even really solve this problem -- still seems like it could cause confusion if 0/len are not conidered boundaries).

- `ByteSlice::utf8_sequence_len(&self, idx)` could behave too many ways -- specifically IDK if it only reads `self[idx]` or if it considers other bytes nearby (e.g. if it's not a leading byte). Making it a top level function only taking a `u8` removes this ambiguity -- reasonably only one thing it could do

Contributor guide

No contributing guide indexed for this repository

Research direction

Begin by reviewing the bstr public API and existing UTF-8 handling; no file or test is named. Resolve whether the API distinguishes leading, continuation, and always-invalid bytes, then define focused coverage for the selected behavior and return shape.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.