`Bytes.toString` drops a leading U+FEFF, so a string does not survive `fromString` and back
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 48
- Forks
- 14
- Avg merge
- 4h 14m
- Merged PRs (30d)
- 1
Description
Found against: gren 0.6.6, gren-lang/core 7.4.2, node 22
Reproduction: in https://github.com/gilramir/gren-bug-reports, 2026-09-14-bytes-tostring-feff; ./run.sh prints the run
below.
Summary
U+FEFF (ZERO WIDTH NO-BREAK SPACE, or the BYTE ORDER MARK / BOM)
is a character, and Bytes.fromString writes it as the three bytes
EF BB BF (UTF-8 encoding). Reading those bytes back loses it when it is the first character:
| call | result | expected |
|---|---|---|
Bytes.length (Bytes.fromString "\u{FEFF}a") |
4 |
4 |
Bytes.toString (Bytes.fromString "\u{FEFF}a") == Just "\u{FEFF}a" |
False |
True |
Bytes.toString of EF BB BF 61 |
Just "a" |
Just "\u{FEFF}a" |
Bytes.toString of EF BB BF |
Just "" |
Just "\u{FEFF}" |
Bytes.toString of EF BB BF EF BB BF |
Just "\u{FEFF}" |
Just "\u{FEFF}\u{FEFF}" |
Bytes.toString of 61 EF BB BF |
Just "a\u{FEFF}" |
Just "a\u{FEFF}" |
Bytes.Decode.string 4 of EF BB BF 61 |
Just "a" |
Just "\u{FEFF}a" |
Bytes.Decode.string calls Bytes.toString on the bytes it read, so any
length-prefixed string field whose value starts with U+FEFF comes back one
character short, and a decoder that checks a re-encoding against its input
refuses it.
The cause is the decoder's options:
function _Bytes_toString(bytes) {
var decoder = new TextDecoder("utf-8", { fatal: true });
TextDecoder, which the WHATWG Encoding Standard
defines, has an ignoreBOM
option, default false, and with it false the decoder treats EF BB BF at the start of its input as a
byte-order mark and removes it. Only the first one is removed, which is why a
second U+FEFF, or one after another character, survives.
Why Bytes.toString shoul keep it
A byte-order mark belongs to the start of a text file or a text stream, and a
library reading one should remove it. core already has that library:
Stream.textDecoder is a TextDecoderStream, and it removes EF BB BF at the
start of the stream and keeps a U+FEFF anywhere after it. Fixing
Bytes.toString does not change that.
Bytes.toString sits below that layer, and it cannot tell whether its bytes
are the start of a file. Most of the time they are not.
Other languages decode UTF-8 the same way: the character is kept, and removing
a mark is something the caller asks for.
decoding EF BB BF 61 |
result |
|---|---|
Python 3.12 bytes.decode("utf-8") |
U+FEFF U+0061 |
Python 3.12 bytes.decode("utf-8-sig"), which asks for removal |
U+0061 |
Go 1.25 string(b) |
U+FEFF U+0061 |
Java 21 new String(b, StandardCharsets.UTF_8) |
U+FEFF U+0061 |
Node 22 Buffer.toString("utf8") |
U+FEFF U+0061 |
Node 22 fs.readFileSync(path, "utf8") |
U+FEFF U+0061 |
new TextDecoder("utf-8", { ignoreBOM: true }) |
U+FEFF U+0061 |
new TextDecoder("utf-8"), what Bytes.toString uses |
U+0061 |
TextDecoder's default comes from the Encoding Standard, which decodes
whole web resources, where a leading mark is expected. That is the same job
Stream.textDecoder does, and not the job Bytes.toString does.
Reproduction
$ node app
Bytes.fromString "\u{FEFF}a" has length 4
Bytes.toString (Bytes.fromString "\u{FEFF}a") == Just "\u{FEFF}a": False
toString [EF BB BF 61]: Just [97]
toString [EF BB BF]: Just []
toString [EF BB BF EF BB BF]: Just [65279]
toString [61 EF BB BF]: Just [97, 65279]
Decode.string 4 [EF BB BF 61]: Just [97]
The lists are the decoded strings' code points.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the _Bytes_toString implementation that constructs TextDecoder("utf-8", { fatal: true }), then compare its behavior with Stream.textDecoder. Run the supplied reproduction and add coverage for leading U+FEFF bytes; done means Bytes.toString and Bytes.Decode.string preserve the leading character while invalid UTF-8 handling remains unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 75/100