ParseOptions::encoding_override should deal with unmappable characters
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 1.6k
- Forks
- 406
- PR merge metrics
- No merged PRs in 30d
Description
The URL spec's "percent-encode after encoding" special-cases the percent-encoding of HTML character references introduced as part of the encoding, such that it's not equivalent to running percent-encode after running encode:
<!DOCTYPE html>
<meta charset="windows-1252">
<script>
const a = document.createElement("a");
a.href = "https://example.com";
a.search = "?ə=\u0259";
console.log(a.search); // ?&%23601;=%26%23601%3B
</script>
While this could be handled by callers of the url crate:
use encoding_rs::{EncoderResult, Encoding, WINDOWS_1252};
use url::Url;
fn url_encoding_override(mut s: &str, encoding: &'static Encoding) -> Vec<u8> {
let mut output = Vec::with_capacity(100);
let mut encoder = encoding.new_encoder();
loop {
match encoder.encode_from_utf8_to_vec_without_replacement(s, &mut output, true) {
(EncoderResult::InputEmpty, _) => break,
(EncoderResult::OutputFull, consumed) => {
output.reserve(100);
s = &s[consumed..];
}
(EncoderResult::Unmappable(ch), consumed) => {
use std::io::Write;
write!(&mut output, "%26%23{}%3B", ch as u32).unwrap();
s = &s[consumed..];
}
};
}
output
}
fn main() {
let url = Url::options()
.encoding_override(Some(&|s| url_encoding_override(s, WINDOWS_1252).into()))
.parse("https://example.com/?&%23601;=ə")
.unwrap();
println!("{:?}", url.query());
}
the fact that Servo gets it wrong makes a good case that this should be rust-url's responsibility.
It's not clear to me what should be the API to replace EncodingOverride, though. Maybe the simplest thing would be to make it an Option<&'static encoding_rs::Encoding>.
cc @annevk, @hsivonen
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing ParseOptions::encoding_override and the EncodingOverride API in the URL parser. Reproduce the windows-1252 example from the issue, then determine an API that preserves unmappable characters as percent-encoded HTML character references and add coverage showing the expected query output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100