JuliaWeb / JuliaWeb/URIs.jl

Incorrect handling of Unicode characters

Open
#41 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Julia
Stars
33
Forks
20
Avg merge
22h 11m
Merged PRs (30d)
1

Description

As @fonsp pointed out in #39, URIs.jl does not technically handle Unicode characters correctly, at least according to RFC 3986. [IETF RFC 3986 Sec. 1.2.1](https://datatracker.ietf.org/doc/html/rfc3986#section-1.2.1) implies that URIs should only contain characters from the US-ASCII charset and should percent-encode additional characters ([RFC 3987](https://datatracker.ietf.org/doc/html/rfc3987) makes this a little more explicit). URIs.jl, however, will accept and work with any string as its input regardless of the underlying character set:

```julia
julia> using URIs

julia> url = URI("https://a/🌟/e")
URI("https://a/🌟/e")

julia> url.path
"/🌟/e"
```

After diving into it for a bit, there seems to be a split in how the standard / canonical library for URI handling works in many other languages. In JavaScript, Go, and Rust, passing in a URI that uses Unicode will either force the URI to be percent-encoded or raise an error:

Javascript

```javascript
>> new URL("https://a/🌟/e").pathname
"/%F0%9F%8C%9F/e"
```

Go

```go
package main

import (
"fmt"
"net/url"
"os"
)

func main() {
url, err := url.Parse("https://a/🌟/e")
if err != nil {
fmt.Fprintf(os.Stderr, "Error parsing url: %s", err)
return
}
fmt.Printf("%s\n", url)
// Prints https://a/%F0%9F%8C%9F/e
}
```

Rust

Rust's `http` crate will actually panic if you try to feed it a Unicode URI at all, e.g.:

```rust
use http::Uri;

fn main() {
let uri = Uri::from_static("https://a/🌟/e");
println!("{}", uri.path());
}
```

```
$ cargo run
Finished dev [unoptimized + debuginfo] target(s) in 0.01s
Running `target/debug/uri`
thread 'main' panicked at 'static str is not valid URI: invalid uri character', /home/kernelmethod/.cargo/registry/src/github.com-1ecc6299db9ec823/http-0.2.7/src/uri/mod.rs:365:23
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```

But this isn't universally the case: in Python and Java, the Unicode encoding is preserved:

Python

```python
>>> from urllib.parse import urlparse
>>> url = urlparse("https://a/🌟/e")
>>> url.path
'/🌟/e'
```

Java

```java
import java.net.*;

class URITesting {
public static void main(String[] args) {
try {
URI url = new URI("https://a/🌟/e");
System.out.printf("path = %s\n", url.getPath());
}
catch (URISyntaxException ex) {
System.out.println(ex);
}
// System.out.println("Hello, World!");
}
}
```

One potential difference between these languages is that Java's [`java.net.URI`](https://docs.oracle.com/javase/7/docs/api/java/net/URI.html) tries to comply with RFC 2936, whereas Python's [`urllib.parse.urlparse`](https://docs.python.org/3/library/urllib.parse.html) seems to try to comply with a mix of standards.

---

In any case, there's a bit of a dilemma here -- this library doesn't quite implement the RFC as specified, which is also an issue that has cropped up in other places, e.g. in the implementation of `normpath` #20 and `joinpath` (related issue: [#18](https://github.com/JuliaWeb/URIs.jl/issues/18#issuecomment-798986903)). As far as this issue is concerned, it seems like there are three ways URIs.jl could go:

1. Percent-encode strings when we generate a URI to ensure compliance to the spec;
2. Implement [RFC 3987](https://datatracker.ietf.org/doc/html/rfc3987) under the hood, which *does* permit Unicode characters; or
3. Keep the library's current behavior and try to specify which parts of URIs.jl comply with which RFCs, similar to what Python does for its `urllib.parse` module.

I would think that option (1) is the most preferable of all of these -- this library says that it implements URIs according to RFC 3986, so it should comply with that RFC.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the URI(...) constructor and trace how Unicode input is preserved or encoded, then compare the related normpath (#20) and joinpath (#18) behavior. Use RFC 3986 and RFC 3987 as the standards references. Done means the project has selected and specified one behavior, applied it consistently, and covered Unicode URI inputs; the issue names no files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.