Escaping/unescaping URIs and elements doesn't work f... | ruby/cgi

还没有人认领这个 Issue。

Escaping (escape, escapeURIComponent) and unescaping (unescape, unescapeURIComponent) work by interpreting the source String as binary and forcing encoding on the output string. However, this completely breaks when encoding isn't single-byte-long in ASCII range, such as UTF-16. All of this applies to all variants of UTF-16 and UTF-32, as far as I'm aware.

For example, escaping technically correctly percent-encodes each octet, but the characters are not encoded in UTF-16, instead the encoding is forced on raw ASCII values:

"\uFEFF".encode(Encoding::UTF_16LE).bytes
=> [255, 254]
CGI.escape_uri_component("\uFEFF".encode(Encoding::UTF_16LE)).bytes
=> [37, 70, 70, 37, 70, 69] # This is %FF%FE in ASCII
CGI.escape_uri_component("\uFEFF".encode(Encoding::UTF_16LE))
=> "\u4625\u2546\u4546" # But this is just 3 unrelated characters!
CGI.escape_uri_component("\uFEFF".encode(Encoding::UTF_16LE)).encoding
=> #<Encoding:UTF-16LE>

On the other hand, unescaping tries to interpret sequential triplets of bytes as a percent-encoded octet, but with the characters in the string all being multibyte, there will always be extra bytes in between, making unescaping completely impossible:

"%FE%FF".encode(Encoding::UTF_16LE).bytes
=> [37, 0, 70, 0, 69, 0, 37, 0, 70, 0, 70, 0]
CGI.unescape_uri_component("%FE%FF".encode(Encoding::UTF_16LE))
=> "%\u0000F\u0000E\u0000%\u0000F\u0000F\u0000" # This is in UTF-8, hence the extra NULs
CGI.unescape_uri_component("%FE%FF".encode(Encoding::UTF_16LE), Encoding::UTF_16LE)
=> "%FE%FF" # This is UTF-16, but not very unescaped

But it may work on accident instead:

what = "┵〥㔷┴䔥㐵┴㐡".encode(Encoding::UTF_16BE)
=> "\u2535\u3025\u3537\u2534\u4525\u3435\u2534\u3421"
CGI.unescapeURIComponent(what)
=> "PWNED!"

Round-tripping looks to be working, but that's mainly by virtue of the methods being inverses of each other. In reality, "escaped" string is definitely not valid:

CGI.unescape_uri_component(CGI.escapeURIComponent("A %$\u1234".encode(Encoding::UTF_16LE)), Encoding::UTF_16LE)
=> "A %$\u1234"
CGI.escapeURIComponent(" %$\u1234".encode(Encoding::UTF_16LE))
=> "\u3225\u2530\u3030\u3225\u2535\u3030\u3225\u2534\u3030\u2534\u3231"
CGI.escapeURIComponent(" %$\u1234".encode(Encoding::UTF_16LE)).encode(Encoding::UTF_8)
=> "㈥┰〰㈥┵〰㈥┴〰┴㈱"

RFC 3986 specifies that data needs to be encoded as a sequence of characters, which in turn can use whatever encoding is suitable. So this method of escaping/unescaping is definitely wrong.

I believe that this issue completely breaks interoperabilty with wchar_t APIs.

[un]escapeHTML seem to use a very different approach, seemingly not being susceptible to this:

CGI.escapeHTML("<html>".encode(Encoding::UTF_16LE))
=> "&lt;html&gt;"
CGI.unescapeHTML(CGI.escapeHTML("<html>".encode(Encoding::UTF_16LE)))
=> "<html>"
CGI.unescapeHTML(CGI.escapeHTML("<html>".encode(Encoding::UTF_16LE))).encoding
=> #<Encoding:UTF-16LE>

[un]escapeElement, on the other hand, try to use UTF-8 strings inside, not working with wide encoding at all:

CGI.escapeElement("<html><body>".encode(Encoding::UTF_16LE), "body".encode(Encoding::UTF_16LE))
# <...>/lib/ruby/3.3.0/cgi/util.rb:187:in `escapeElement': incompatible character encodings: UTF-8 and UTF-16LE (Encoding::CompatibilityError)
CGI.unescapeElement("<html>&lt;body&gt;".encode(Encoding::UTF_16LE), "body".encode(Encoding::UTF_16LE))
# <...>/lib/ruby/3.3.0/cgi/util.rb:207:in `unescapeElement': incompatible character encodings: UTF-8 and UTF-16LE (Encoding::CompatibilityError)

(This session was on Ruby 3.3, but the same happens in 3.4 and I believe that there is no difference in the extracted gem either.)

从 lib/cgi/escape.rb 以及 lib/cgi/util.rb 中的 escapeElement/unescapeElement 入口开始，复现 issue 中的 UTF-16 和 UTF-32 示例。跟踪每个方法如何处理已编码的字节和元素名称；对于报告的宽编码，如果转义、反转义和元素操作都能正确运行且不会出现编码兼容性错误，则表示完成。

由索引模型根据 Issue 内容生成。

Escaping/unescaping URIs and elements doesn't work for UTF-16/32

描述

贡献指南

从这里开始

调研方向

评估

Escaping/unescaping URIs and elements doesn't work for UTF-16/32

描述

贡献指南

从这里开始

调研方向

评估

把新 issue 发到你的邮箱