gren-lang / gren-lang/core

`firstIndexOf` can return an index that `slice` cannot use: what it found is half a character

Open
#150 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
JavaScript
Stars
48
Forks
14
Avg merge
4h 14m
Merged PRs (30d)
1

Description

This is related to #148 but different. It's another bug related to UTF-16 surrogates.

Found against: gren 0.6.6, gren-lang/core 7.4.2, node 22
Reproduction: https://github.com/gilramir/gren-bug-reports 2026-09-12-string-half-surrogate; ./run.sh prints the examples below.

Summary

This is a typical program: find something, then slice at what was found:

when String.firstIndexOf sub str is
    Just i ->
        String.slice i (i + String.count sub) str

    Nothing ->
        ""

For one kind of occurence it does not return sub, and no index would have made it
work
.

𝄞 is U+1D11E, one character, stored on JavaScript as two UTF-16 code units.
String.sliceUnits can cut between them, which gives a string that is half of a
𝄞:

clef       = "𝄞ab"                       -- three characters: 𝄞, a, b
                                         -- the 𝄞 is U+1D11E, stored as the two
                                         -- code units 0xD834 and 0xDD1E
secondUnit = String.sliceUnits 1 2 clef  -- just the 0xDD1E

String.firstIndexOf secondUnit clef      -- Just 1
String.slice 1 2 clef                    -- "a"

clef's characters are 𝄞, a and b. secondUnit is not one of them, so
there is no character position for firstIndexOf to report: Just 0 would mean
the 𝄞 and Just 1 means the a, and neither is what matched. Nothing is
the only answer that is not wrong.
lastIndexOf and indices do the same
thing.

This is not #148. There the index is a real position reported in the wrong unit,
and converting it is the whole fix. Here there is no position to convert to — so
whoever fixes #148 has to decide how to handle this case too.

Terms

From String's own module documentation: a code unit is "the smallest
primitive value of a string", 16 bits in Gren; a code point "represents a
unicode character", and takes one code unit or two. count, slice and
toArray work in code points, unitLength and sliceUnits in code units, and
"unless otherwise noted, all functions in this module deal with code points" —
none of the six named above is noted.

One thing that documentation leaves out matters here: a surrogate
(U+D800–U+DFFF, the values reserved for building pairs) is a code point too. So
secondUnit is a perfectly ordinary one-character StringString.count of
it is 1 — and it cannot be dismissed as malformed input.

Nor does it have to come from sliceUnits. Anything that turns a code point into
a String can build it; checked against 7.4.2: sliceUnits, getUnit,
foldlUnits, a \u{DD1E} escape in a literal, and
Char.fromCode 0xDD1E |> String.fromChar, which is the fromCode row in the
table below.

Reproduction

./run.sh produces the rows below.

clef       = "𝄞ab"                                   -- 𝄞, a, b
clefChar   = "𝄞"                                     -- U+1D11E = 0xD834 0xDD1E
firstUnit  = String.sliceUnits 0 1 clef              -- 0xD834, the first half
secondUnit = String.sliceUnits 1 2 clef              -- 0xDD1E, the second half
fromCode   = String.fromChar (Char.fromCode 0xDD1E)  -- secondUnit again, built
                                                     -- without any *Units call
call result expected
String.firstIndexOf secondUnit clef Just 1 Nothing
String.firstIndexOf firstUnit clef Just 0 Nothing
String.lastIndexOf secondUnit clef Just 1 Nothing
String.indices secondUnit clef [1] []
String.firstIndexOf fromCode clef Just 1 Nothing
String.contains secondUnit clef True False
String.startsWith firstUnit clef True False
String.endsWith secondUnit clefChar True False
String.contains fromCode clef True False
String.contains clefChar clef True True
String.firstIndexOf secondUnit secondUnit Just 0 Just 0

The last two rows are already right, and they are there to bound the fix. The
final one is why the fix has to ask where an occurrence falls in the string
being searched
, and not what is being searched for. secondUnit on its own is
a whole one-character string, so finding it inside itself at index 0 is correct;
what is wrong in the rows above is where it is found, inside a character of
clef rather than between two of them. A rule that rejected any search string
containing a surrogate would get that last row wrong.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running ./run.sh from the linked gren-bug-reports reproduction 2026-09-12-string-half-surrogate and inspect the implementations of String.firstIndexOf, lastIndexOf, indices, contains, startsWith, and endsWith. Compare their behavior with the reproduction table, especially matches inside the surrogate pair versus secondUnit searched within itself. Done means the listed expected results hold without breaking the already-correct final two rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.