gren-lang / gren-lang/compiler

`\u{FFFF}` in a string literal produces the wrong result for a couple different reasons

Open Beginner friendly
#384 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Haskell
Stars
503
Forks
29
PR merge metrics
No merged PRs in 30d

Description

Package: gren-lang/compiler (the Haskell-based compiler)
File: compiler/src/Gren/String.hs
Version observed: gren 0.6.6 (via devbox gren 0.6)

Summary

A \u{FFFF} escape in a string literal compiles to the two code units
U+D7FF U+DFFF instead of the single character U+FFFF. Every other code
point tested is correct, including U+FFFE just below it and U+10000 just
above. Note that these 2 code units, U+D7FF U+DFFF, are not even valid surrogate pairs.

U+D7FF is not a surrogate. It sits one below U+D800, where the high-surrogate range starts (D71/D74) — it's an ordinary BMP code point, reserved in Hangul Jamo Extended-B. It's a perfectly legal Unicode scalar value that any string may contain.

Reproduction

import Array
import Char
import String

codes : Array Int
codes =
    Array.map Char.toCode
        (String.toArray "\u{FFFC}|\u{FFFD}|\u{FFFE}|\u{FFFF}|\u{10000}|\u{10001}")

Expected:

[ 65532, 124, 65533, 124, 65534, 124, 65535, 124, 65536, 124, 65537 ]

Actual:

[ 65532, 124, 65533, 124, 65534, 124, 55295, 57343, 124, 65536, 124, 65537 ]
                                      ^^^^^^^^^^^^

55295 is 0xD7FF and 57343 is 0xDFFF. These are wrong. Note
that 0xD7FF is not a surrogate, so the result is a well-formed string
containing two wrong characters.

Cause

Two off-by-one comparisons, both in compiler/src/Gren/String.hs:

-- line 78
chunkToWidth chunk =
  case chunk of
    ...
    CodePoint c -> if c < 0xFFFF then 6 else 12
                        -- ^ should be c < 0x10000

-- line 99
        CodePoint code ->
          if code < 0xFFFF
                 -- ^ should be code < 0x10000
            then do
              writeCode mba offset code
              ...
            else do
              let (hi, lo) = divMod (code - 0x10000) 0x400
              writeCode mba (offset) (hi + 0xD800)
              writeCode mba (offset + 6) (lo + 0xDC00)

0xFFFF is the last code point in the BMP and needs no surrogate pair, but
< 0xFFFF sends it down the pair branch anyway. There, divMod (0xFFFF - 0x10000) 0x400 is divMod (-1) 0x400, which Haskell floors to (-1, 1023), so:

  • high surrogate = -1 + 0xD800 = 0xD7FF
  • low surrogate = 1023 + 0xDC00 = 0xDFFF

The two sites agree with each other, so the byte array is the right length and
nothing is corrupted beyond the two code units -- which is probably why this has
not been noticed.

Where the spec says the boundary is

The threshold is 0x10000, not 0xFFFF. U+FFFF is the last code point of the
Basic Multilingual Plane and is representable in a single UTF-16 code unit; only
the supplementary planes need a pair.

The Unicode Standard, section 3.9.2, definition D91 (UTF-16 encoding form) --
the normative statement:

Code points in the range U+0000 to U+FFFF are represented by a single 16-bit
code unit equal to the code point value. Code points in the range U+10000 to
U+10FFFF are represented by two code units, the first from the range U+D800 to
U+DBFF and the second from U+DC00 to U+DFFF.

RFC 2781, section 2.1, "Encoding UTF-16" -- the same rule written as the
algorithm this code is implementing, and the shortest way to state the fix:

If U < 0x10000, encode U as a 16-bit unsigned integer and terminate.

Impact

Small , but I hit it. U+FFFF is a noncharacter and rarely written on purpose, so
most programs will never hit it. It shows up when a program is exercising the
edges of the code point space deliberately -- which is how I found it.

This was found while porting the official TOML test suite:
valid/string/quoted-unicode.toml contains <ffff> among a row of boundary
code points, and the generated Gren fixture for it silently became a different
string, so a correct parser appeared to fail.

Anything constructing the character at runtime is fine: Char.fromCode 0xFFFF
goes through String.fromCodePoint in the JS kernel and is correct. It is only
the compile-time escape.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in compiler/src/Gren/String.hs, at chunkToWidth and the CodePoint branch that writes UTF-16 units. Run the supplied boundary reproduction and compare it with the expected code points, then verify the TOML valid/string/quoted-unicode.toml fixture. Done means U+FFFF is emitted as one code unit while U+10000 and later still use surrogate pairs.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell
Domain
compilers
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.