mathiasbynens / mathiasbynens/utf8.js

The README should probably mention that output only looks like UTF-8, but isn't actual UTF-8

Open
#42 4 comments 2 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
562
Forks
113
PR merge metrics
No merged PRs in 30d

Description

This module encodes a string to *look like* a UTF-8 string, which may be used for online UTF-8 demos, but as far as bytes are concerned, which is important for hashing, etc, the output is not actually UTF-8.

Take your README example with the copyright character.
```javascript
// U+00A9 COPYRIGHT SIGN; see http://codepoints.net/U+00A9
utf8.encode('\xA9');
// → '\xC2\xA9'
```
, each of `\xXX` sequences in JavaScript produces a standalone code point, so `\xA9` natively will be represented as UTF-16 in JavaScript (well, UCS2, really), which can be seen here:
```javascript
console.log(Buffer.from('\xA9', 'utf16le')
```
, which yields a code point U+00A9 in little endian notation:
```

```
This is how one can generate an actual UTF-8 sequence. Either of these will work (the default encoding is UTF-8):
```javascript
console.log(Buffer.from('\xA9'))
console.log(Buffer.from('\xA9', 'utf8'))
```
, and will produce UTF-8 bytes, which are good for hashing and other uses where it matters:
```

```
For example, this yields the correct MD5 hash of the `\xA9` represented as UTF-8 because `update` does the same transformation `Buffer.from` uses:
```javascript
console.log(crypto.createHash('md5').update('\xA9').digest('hex'))
```
, which is `a541ecda3d4c67f1151cad5075633423`. This will *not* produce the correct hash:
```javascript
console.log(crypto.createHash('md5').update(utf8.encode('\xA9')).digest('hex'))
```
, which actually hashes `` and yields `1b4c0262ce2f67450c4ecb3026ab1350`.

This fooled even Microsoft, who referenced `utf8` in their docs, which only works because their input is always ASCII, which makes `utf8.encode()` a no-op.

https://docs.microsoft.com/en-us/rest/api/eventhub/generate-sas-token#nodejs

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the README's copyright-character example and compare it with the Buffer and hashing examples described in the issue. Update the documentation to distinguish output that only looks like UTF-8 from actual UTF-8 bytes, and ensure the example explains the hashing consequence shown in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
documentation
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.