BurntSushi / BurntSushi/encoding_rs_io
Always transcode Utf8
- Dominant language
- Rust
- Stars
- 30
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
I'm using encoding_rs_io to make a stream of always valid utf8 because invalid utf8 is not handled upstream. The way the options are laid out at present it seems there's no way to force transcoding to occur if there is no BOM in the file. I think I found a way to do it by layering multiple DecodeReaderBytes over each other, but I'm unsure that it works in all cases and a little dismayed that it requires multiple layers instead of just having an option to force transcoding.
Here's the code I have today:
```
pub fn new_utf8_reader(data: &[u8]) -> impl Read + '_ {
let cursor = Cursor::new(data);
// The first layer has utf8-passthrough, and the second
// no passthrough but an explicit encoding. This unexpected
// chain was concocted to handle the case where the file has
// no BOM and is encoded with something other than utf8 or
// contains invalid utf-8 characters. Basically, this
// forces transcoding.
// When there is a non UFT-8 BOM the first layer will transcode to UTF-8. (so will the second, redundantly)
// When there is no BOM or a UTF-8 BOM the second layer will transcode to UTF-8.
let uncorrected = DecodeReaderBytesBuilder::new()
.utf8_passthru(true)
.build(cursor);
DecodeReaderBytesBuilder::new()
.encoding(Some(UTF_8))
.strip_bom(true)
.build(uncorrected)
}
```
Is this the best way to force transcoding to utf8 in the presence of unknown data (which may or may not contain a BOM, and may or may not be valid) given the API today?
I don't think I'm the only one with this problem. It took some time to figure out an answer. Would it be worth it to do one of the following...
1. Add this usage to the documentation as a recipe?
2. Introduce a new 'force transcoding' option?
3. Add a factory function that does this?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the DecodeReaderBytesBuilder usage shown in the issue and inspect the repository's existing API and documentation. Compare the recipe, a force-transcoding option, and a factory function against the unknown-data and BOM cases described. Done means the supported approach is clearly documented or an agreed API change is specified and validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100