string operations should work an characters not bytes

Describe the bug

All string operations should work an characters and not bytes.

Reasons:

• According to the documentation[1] a string is "a series of characters that represents text" (emphasis mine). Obviously, characters are not bytes.

• Using bytes instead of characters is unexpected:

  >  "hällo" | str length
  6
  

  However, "hällo" (obviously) has five characters.

• It really is unexpected:

  > "hällo" | str length
  7
  

  For the uninitiated: This uses a (U+0061) + ◌̈ (U+0308) instead of ä (U+00E4) which as also a valid Unicode representation of "ä".

• This is the behaviour as learned in elementary school for users of non-ASCII languages (and probably any language) [I don't speak every language of the world, so I might be wrong here]. This are almost all languages since many have non-ASCII characters or no ASCII letters at all.

  Some data argument:
  If you look at [2] this is the mayority of humans (and probably also users of nushell though I have no data on that)!
  Only 390 million out of 8 billion, about 5%, are native English. (does not account for native English speakers who must deal with foreign languages)

• Characters are for humans, bytes are for computers. The UX, API, etc should be for humans not computers[4].

• This also easily leads to programming errors when dealing with user input including filenames:

  # Shorten the path to two characters in the prompt
  > let path = "hällo/býe"
  > $path | split row "/" | str substring 0..1 | each {|s| $s ++ "…"} | str join '/'
  h�…/b�…
  

  I customized my prompt and I always without exception operated on characters not bytes, for shortening, padding, etc.

• Note that the above code is broken as the string is not a valid UTF-8 encoded string which according to [3] is required:

  Internally strings are UTF-8 encoded

  Taking that requirement to the extreme, then even the type of str substring : string -> string is wrong as "hällo" | str substring 0..1 is not a string. The correct type would be string -> binary or similar.

• str substring is called substring and not subbytes

• string being stored as a sequence of bytes is an implementation detail.

All(?) operations can be persuaded to behave human friendly by supplying the --grapheme-clusters flag which operates on "visible characters". This option should be the default for all string-related operations. For the rare use case that byte operations are required either a into binary can (and should) be used or an new flag --as-binary-sequence could be implemented for the string commands.

If there is fear of seriously breaking scripts, there could be a transition period where this change is hidden behind a experimental option.

Side note: There is also an argument to be made to use Unicode code-points instead of graphemes but so far I've only found the split chars command to use code-points. Having said that, code-point and grapheme are both strictly better than bytes.

[1] https://www.nushell.sh/book/types_of_data.html#text-strings
[2] https://en.wikipedia.org/wiki/List_of_languages_by_number_of_native_speakers
[3] https://www.nushell.sh/lang-guide/chapters/types/basic_types/string.html
[4] I know that some programming languages, e.g. C, claim to have a "string" type but its just a "sequence of bytes" type.

How to reproduce

See examples above.

Expected behavior

• Human friendliness
• Openness for other cultures and languages.
• Also: doing the obvious and not the easy. ;-)

Configuration

0.106.1

Start by inventorying the string-related operations shown in the examples, including str length, str substring, split chars, and the existing --grapheme-clusters option. Compare their current behavior with the documented UTF-8 string model and the provided composed and decomposed examples. Done means the intended default character behavior and any explicit binary behavior are defined consistently across the affected operations.