BurntSushi / BurntSushi/encoding_rs_io

[RFE] User-provided encoding detection function

Open
#13 1 comment 0 reactions 0 assignees View on GitHub
question
Dominant language
Rust
Stars
30
Forks
10
PR merge metrics
No merged PRs in 30d

Description

As an example, the XML specification recommends a special encoding detection scheme in cases where the BOM doesn't exist: https://www.w3.org/TR/xml11/#sec-guessing

In the event that a single-byte, ascii-compatible encoding is being used, you're supposed to inspect the XML declaration to determine which specific encoding to use.

My initial thoughts about implementing this are: the code could just read a full buffer of data (instead of using `BomPeeker`) and pass a reference to this buffer directly to the encoding detection functions (`Encoding::for_bom(&[u8])` and / or a user-provided one), adjusting `self.pos` if necessary to ignore the BOM.

That would simplify the code at the same time, with the caveat that the user must ensure that the buffer is sufficient for the detection schemes (e.g., minimum 3 bytes for BOM detection), but that feels like a reasonable restriction?

It could look something like this:

```Rust
pub fn xml_detect_encoding(bytes: &[u8]) -> Option<&'static Encoding> {
match bytes {
_ if bytes.starts_with(&[0x00, b'<', 0x00, b'?']) => Some((UTF_16BE, 0)), // Some BE encoding, for example, UTF-16 or ISO-10646-UCS-2
_ if bytes.starts_with(&[b'<', 0x00, b'?', 0x00]) => Some((UTF_16LE, 0)), // Some LE encoding, for example, UTF-16 or ISO-10646-UCS-2
_ if bytes.starts_with(&[b'<', b'?', b'x', b'm']) => { // Some ASCII compatible
unimplemented!(r#"parse from the XML 'encoding' tag e.g. "#);
}
_ => None,
}
}

let f = File::open("inputdata.xml")?;
let mut rdr = DecodeReaderBytes::new(f);
rdr.detect_encoding_with(xml_detect_encoding)?;
assert_eq!(rdr.encoding(), UTF_16LE);
```

It could be a `DecodeReaderBytesBuilder` option instead of an explicit function call, but it feels like a slightly different category from the others.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the DecodeReaderBytes flow, including BomPeeker and Encoding::for_bom, then review how a user-provided detector or DecodeReaderBytesBuilder option would fit. The XML detection example in the issue defines the intended behavior; done means the API supports that detection path, handles BOM position correctly, and has coverage for the described cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.