gleam-lang / gleam-lang/stdlib

Gleam `string` module said erroneous information about string encoding

Open
#931 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Gleam
Stars
710
Forks
225
Avg merge
5h 52m
Merged PRs (30d)
3

Description

In the current `string` module you can read this

> //// Strings in Gleam are UTF-8 binaries. They can be written in your code as
> //// text surrounded by `"double quotes"`.

This text was originally written for gleam v0.8. (Apr 28 2020)
This was around 1 year before gleam v0.16 arrived and introduce JavaScript target.

And so since then nobody did updated this message and so people could be confused by thinking Gleam encode internally every string into binary UTF-8 which is not true since on Erlang they are just regular `Bitstring` (UTF8) and on JS it's just the String type of JS which is UTF16.

Now another issue in this module is function like `byte_size`.
This function is made to return the number of byte of a String in Gleam.
However nowhere it said this function return the number of byte of an encoded UTF8 text.
So again people could thing it will on JavaScript return UTF16 which is not the case since it will use `TextEncoder` to convert it into UTF8 and then count.
We should probably mentioned that fact so people don't think it's related to the String size of the platform.

(Just this bit is speculation and not really useful but I think the reason it's not UTF16 for JS was because this function existed before v0.16 and so when added needed to follow the same behavior of Erlang to not break existing program)

Maybe I miss other point where it would be useful to specify if the encoding is UTF8 converted or follow the platform encoding but this is what I know right now.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.