gren-lang / gren-lang/compiler
`\u{FFFF}` in a string literal produces the wrong result for a couple different reasons
Nobody has claimed this yet.
- Dominant language
- Haskell
- Stars
- 503
- Forks
- 29
- PR merge metrics
- No merged PRs in 30d
Description
Package: gren-lang/compiler (the Haskell-based compiler)
File: compiler/src/Gren/String.hs
Version observed: gren 0.6.6 (via devbox gren 0.6)
Summary
A \u{FFFF} escape in a string literal compiles to the two code units
U+D7FF U+DFFF instead of the single character U+FFFF. Every other code
point tested is correct, including U+FFFE just below it and U+10000 just
above. Note that these 2 code units, U+D7FF U+DFFF, are not even valid surrogate pairs.
U+D7FF is not a surrogate. It sits one below U+D800, where the high-surrogate range starts (D71/D74) — it's an ordinary BMP code point, reserved in Hangul Jamo Extended-B. It's a perfectly legal Unicode scalar value that any string may contain.
Reproduction
import Array
import Char
import String
codes : Array Int
codes =
Array.map Char.toCode
(String.toArray "\u{FFFC}|\u{FFFD}|\u{FFFE}|\u{FFFF}|\u{10000}|\u{10001}")
Expected:
[ 65532, 124, 65533, 124, 65534, 124, 65535, 124, 65536, 124, 65537 ]
Actual:
[ 65532, 124, 65533, 124, 65534, 124, 55295, 57343, 124, 65536, 124, 65537 ]
^^^^^^^^^^^^
55295 is 0xD7FF and 57343 is 0xDFFF. These are wrong. Note
that 0xD7FF is not a surrogate, so the result is a well-formed string
containing two wrong characters.
Cause
Two off-by-one comparisons, both in compiler/src/Gren/String.hs:
-- line 78
chunkToWidth chunk =
case chunk of
...
CodePoint c -> if c < 0xFFFF then 6 else 12
-- ^ should be c < 0x10000
-- line 99
CodePoint code ->
if code < 0xFFFF
-- ^ should be code < 0x10000
then do
writeCode mba offset code
...
else do
let (hi, lo) = divMod (code - 0x10000) 0x400
writeCode mba (offset) (hi + 0xD800)
writeCode mba (offset + 6) (lo + 0xDC00)
0xFFFF is the last code point in the BMP and needs no surrogate pair, but
< 0xFFFF sends it down the pair branch anyway. There, divMod (0xFFFF - 0x10000) 0x400 is divMod (-1) 0x400, which Haskell floors to (-1, 1023), so:
- high surrogate =
-1 + 0xD800=0xD7FF - low surrogate =
1023 + 0xDC00=0xDFFF
The two sites agree with each other, so the byte array is the right length and
nothing is corrupted beyond the two code units -- which is probably why this has
not been noticed.
Where the spec says the boundary is
The threshold is 0x10000, not 0xFFFF. U+FFFF is the last code point of the
Basic Multilingual Plane and is representable in a single UTF-16 code unit; only
the supplementary planes need a pair.
The Unicode Standard, section 3.9.2, definition D91 (UTF-16 encoding form) --
the normative statement:
Code points in the range U+0000 to U+FFFF are represented by a single 16-bit
code unit equal to the code point value. Code points in the range U+10000 to
U+10FFFF are represented by two code units, the first from the range U+D800 to
U+DBFF and the second from U+DC00 to U+DFFF.
RFC 2781, section 2.1, "Encoding UTF-16" -- the same rule written as the
algorithm this code is implementing, and the shortest way to state the fix:
If U < 0x10000, encode U as a 16-bit unsigned integer and terminate.
Impact
Small , but I hit it. U+FFFF is a noncharacter and rarely written on purpose, so
most programs will never hit it. It shows up when a program is exercising the
edges of the code point space deliberately -- which is how I found it.
This was found while porting the official TOML test suite:
valid/string/quoted-unicode.toml contains <ffff> among a row of boundary
code points, and the generated Gren fixture for it silently became a different
string, so a correct parser appeared to fail.
Anything constructing the character at runtime is fine: Char.fromCode 0xFFFF
goes through String.fromCodePoint in the JS kernel and is correct. It is only
the compile-time escape.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in compiler/src/Gren/String.hs, at chunkToWidth and the CodePoint branch that writes UTF-16 units. Run the supplied boundary reproduction and compare it with the expected code points, then verify the TOML valid/string/quoted-unicode.toml fixture. Done means U+FFFF is emitted as one code unit while U+10000 and later still use surrogate pairs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100