emberjs / emberjs/simple-html-tokenizer

EntityParser can't handle encoded emoji

Open
#67 2 comments 1 reaction 2 assignees Claimed by @wycats View on GitHub
Dominant language
TypeScript
Stars
85
Forks
34
PR merge metrics
No merged PRs in 30d

Description

While the tokenizer will gracefully decode most encoded characters:

```
⇒ node
> var Tokenizer = require( 'simple-html-tokenizer' );
undefined
> Tokenizer.tokenize( '&' )[ 0 ].chars === '&'
true
```

It doesn't handle characters whose encodings exceed 16 bits (e.g. emoji):

```
⇒ node
> var Tokenizer = require( 'simple-html-tokenizer' );
undefined
> Tokenizer.tokenize( '😅' )[ 0 ].chars === '😅'
false
```

It may be that `EntityParser` should use `String.fromCodePoint` in place of `String.fromCharCode` instead, or an equivalent polyfill?

Related:

- https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/fromCodePoint
- https://thekevinscott.com/emojis-in-javascript/

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.