gren-lang / gren-lang/core

`Bytes.toString` drops a leading U+FEFF, so a string does not survive `fromString` and back

Open Beginner friendly
#155 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
JavaScript
Stars
48
Forks
14
Avg merge
4h 14m
Merged PRs (30d)
1

Description

Found against: gren 0.6.6, gren-lang/core 7.4.2, node 22
Reproduction: in https://github.com/gilramir/gren-bug-reports, 2026-09-14-bytes-tostring-feff; ./run.sh prints the run
below.

Summary

U+FEFF (ZERO WIDTH NO-BREAK SPACE, or the BYTE ORDER MARK / BOM)
is a character, and Bytes.fromString writes it as the three bytes
EF BB BF (UTF-8 encoding). Reading those bytes back loses it when it is the first character:

call result expected
Bytes.length (Bytes.fromString "\u{FEFF}a") 4 4
Bytes.toString (Bytes.fromString "\u{FEFF}a") == Just "\u{FEFF}a" False True
Bytes.toString of EF BB BF 61 Just "a" Just "\u{FEFF}a"
Bytes.toString of EF BB BF Just "" Just "\u{FEFF}"
Bytes.toString of EF BB BF EF BB BF Just "\u{FEFF}" Just "\u{FEFF}\u{FEFF}"
Bytes.toString of 61 EF BB BF Just "a\u{FEFF}" Just "a\u{FEFF}"
Bytes.Decode.string 4 of EF BB BF 61 Just "a" Just "\u{FEFF}a"

Bytes.Decode.string calls Bytes.toString on the bytes it read, so any
length-prefixed string field whose value starts with U+FEFF comes back one
character short, and a decoder that checks a re-encoding against its input
refuses it.

The cause is the decoder's options:

function _Bytes_toString(bytes) {
  var decoder = new TextDecoder("utf-8", { fatal: true });

TextDecoder, which the WHATWG Encoding Standard
defines, has an ignoreBOM
option, default false, and with it false the decoder treats EF BB BF at the start of its input as a
byte-order mark and removes it. Only the first one is removed, which is why a
second U+FEFF, or one after another character, survives.

Why Bytes.toString shoul keep it

A byte-order mark belongs to the start of a text file or a text stream, and a
library reading one should remove it. core already has that library:
Stream.textDecoder is a TextDecoderStream, and it removes EF BB BF at the
start of the stream and keeps a U+FEFF anywhere after it. Fixing
Bytes.toString does not change that.

Bytes.toString sits below that layer, and it cannot tell whether its bytes
are the start of a file. Most of the time they are not.

Other languages decode UTF-8 the same way: the character is kept, and removing
a mark is something the caller asks for.

decoding EF BB BF 61 result
Python 3.12 bytes.decode("utf-8") U+FEFF U+0061
Python 3.12 bytes.decode("utf-8-sig"), which asks for removal U+0061
Go 1.25 string(b) U+FEFF U+0061
Java 21 new String(b, StandardCharsets.UTF_8) U+FEFF U+0061
Node 22 Buffer.toString("utf8") U+FEFF U+0061
Node 22 fs.readFileSync(path, "utf8") U+FEFF U+0061
new TextDecoder("utf-8", { ignoreBOM: true }) U+FEFF U+0061
new TextDecoder("utf-8"), what Bytes.toString uses U+0061

TextDecoder's default comes from the Encoding Standard, which decodes
whole web resources, where a leading mark is expected. That is the same job
Stream.textDecoder does, and not the job Bytes.toString does.

Reproduction

$ node app
Bytes.fromString "\u{FEFF}a" has length 4
Bytes.toString (Bytes.fromString "\u{FEFF}a") == Just "\u{FEFF}a": False
toString [EF BB BF 61]:          Just [97]
toString [EF BB BF]:             Just []
toString [EF BB BF EF BB BF]:    Just [65279]
toString [61 EF BB BF]:          Just [97, 65279]
Decode.string 4 [EF BB BF 61]:   Just [97]

The lists are the decoded strings' code points.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the _Bytes_toString implementation that constructs TextDecoder("utf-8", { fatal: true }), then compare its behavior with Stream.textDecoder. Run the supplied reproduction and add coverage for leading U+FEFF bytes; done means Bytes.toString and Bytes.Decode.string preserve the leading character while invalid UTF-8 handling remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.