josdejong / josdejong/svelte-jsoneditor

Unicode and invisible characters

Open
#409 2 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted
Dominant language
TypeScript
Stars
1.3k
Forks
148
PR merge metrics
No merged PRs in 30d

Description

[Samples](https://jsoneditoronline.org/#left=cloud.181b0640d5c34fe7b2b2cd1f7dc81927)

When working with unicode (that is, almost any text), you need to remember many things. Here are a few of them:
- Invisible characters
[Invisible characters](https://invisible-characters.com/) can behave differently on different devices, browsers, and fonts. They are usually invisible, but they still take up space.
```js
"឴" != "";
"_឴_" != "__";
```
That's how they are highlighted in the VS Code:
![image](https://github.com/josdejong/svelte-jsoneditor/assets/61969794/d5c47a89-dc4f-4f42-b500-f4628a875be8)

- Combining character and cursed strings
The display of the combining character depends on many factors. They can often display strangely and break the interface and styles.

This is how they are currently displayed in the editor:
![image](https://github.com/josdejong/svelte-jsoneditor/assets/61969794/f011f114-0dda-4824-a0b7-779be4f4e96d)

That's how they are displayed in the VS Code:
![image](https://github.com/josdejong/svelte-jsoneditor/assets/61969794/c3d0b2e0-c983-4962-b4af-a55fe8d36a8f)

- Surrogate couples and normalization
https://en.wikipedia.org/wiki/Unicode_equivalence
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/normalize
```js
"_._".normalize(); // "_._"
"_._".normalize("NFC"); // "_._"
"_._".normalize("NFD"); // "_._"
"_._".normalize("NFKC"); // "_._"
"_._".normalize("NFKD"); // "_._"

const name1 = "\u0041\u006d\u00e9\u006c\u0069\u0065";
const name2 = "\u0041\u006d\u0065\u0301\u006c\u0069\u0065";
name1 != name2; // "Amélie" != "Amélie"
name1.length != name2.length

const name1NFC = name1.normalize("NFC");
const name2NFC = name2.normalize("NFC");
name1NFC == name2NFC; // "Amélie" == "Amélie"
name1NFC.length == name2NFC.length
```
Before and after formatting:
![image](https://github.com/josdejong/svelte-jsoneditor/assets/61969794/43a236fa-af2c-4839-bbf7-ad4318713827)

---

Everything seems to be fine with this in the editor now.
I suggest:
- highlight invisible characters
- automatically normalize and decode all strings when pasting or formatting

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.