BurntSushi / BurntSushi/encoding_rs_io
[RFE] User-provided encoding detection function
- Dominant language
- Rust
- Stars
- 30
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
As an example, the XML specification recommends a special encoding detection scheme in cases where the BOM doesn't exist: https://www.w3.org/TR/xml11/#sec-guessing
In the event that a single-byte, ascii-compatible encoding is being used, you're supposed to inspect the XML declaration to determine which specific encoding to use.
My initial thoughts about implementing this are: the code could just read a full buffer of data (instead of using `BomPeeker`) and pass a reference to this buffer directly to the encoding detection functions (`Encoding::for_bom(&[u8])` and / or a user-provided one), adjusting `self.pos` if necessary to ignore the BOM.
That would simplify the code at the same time, with the caveat that the user must ensure that the buffer is sufficient for the detection schemes (e.g., minimum 3 bytes for BOM detection), but that feels like a reasonable restriction?
It could look something like this:
```Rust
pub fn xml_detect_encoding(bytes: &[u8]) -> Option<&'static Encoding> {
match bytes {
_ if bytes.starts_with(&[0x00, b'<', 0x00, b'?']) => Some((UTF_16BE, 0)), // Some BE encoding, for example, UTF-16 or ISO-10646-UCS-2
_ if bytes.starts_with(&[b'<', 0x00, b'?', 0x00]) => Some((UTF_16LE, 0)), // Some LE encoding, for example, UTF-16 or ISO-10646-UCS-2
_ if bytes.starts_with(&[b'<', b'?', b'x', b'm']) => { // Some ASCII compatible
unimplemented!(r#"parse from the XML 'encoding' tag e.g. "#);
}
_ => None,
}
}
let f = File::open("inputdata.xml")?;
let mut rdr = DecodeReaderBytes::new(f);
rdr.detect_encoding_with(xml_detect_encoding)?;
assert_eq!(rdr.encoding(), UTF_16LE);
```
It could be a `DecodeReaderBytesBuilder` option instead of an explicit function call, but it feels like a slightly different category from the others.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the DecodeReaderBytes flow, including BomPeeker and Encoding::for_bom, then review how a user-provided detector or DecodeReaderBytesBuilder option would fit. The XML detection example in the issue defines the intended behavior; done means the API supports that detection path, handles BOM position correctly, and has coverage for the described cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100