mathiasbynens / mathiasbynens/utf8.js
The README should probably mention that output only looks like UTF-8, but isn't actual UTF-8
- Dominant language
- JavaScript
- Stars
- 562
- Forks
- 113
- PR merge metrics
- No merged PRs in 30d
Description
This module encodes a string to *look like* a UTF-8 string, which may be used for online UTF-8 demos, but as far as bytes are concerned, which is important for hashing, etc, the output is not actually UTF-8.
Take your README example with the copyright character.
```javascript
// U+00A9 COPYRIGHT SIGN; see http://codepoints.net/U+00A9
utf8.encode('\xA9');
// → '\xC2\xA9'
```
, each of `\xXX` sequences in JavaScript produces a standalone code point, so `\xA9` natively will be represented as UTF-16 in JavaScript (well, UCS2, really), which can be seen here:
```javascript
console.log(Buffer.from('\xA9', 'utf16le')
```
, which yields a code point U+00A9 in little endian notation:
```
```
This is how one can generate an actual UTF-8 sequence. Either of these will work (the default encoding is UTF-8):
```javascript
console.log(Buffer.from('\xA9'))
console.log(Buffer.from('\xA9', 'utf8'))
```
, and will produce UTF-8 bytes, which are good for hashing and other uses where it matters:
```
```
For example, this yields the correct MD5 hash of the `\xA9` represented as UTF-8 because `update` does the same transformation `Buffer.from` uses:
```javascript
console.log(crypto.createHash('md5').update('\xA9').digest('hex'))
```
, which is `a541ecda3d4c67f1151cad5075633423`. This will *not* produce the correct hash:
```javascript
console.log(crypto.createHash('md5').update(utf8.encode('\xA9')).digest('hex'))
```
, which actually hashes `` and yields `1b4c0262ce2f67450c4ecb3026ab1350`.
This fooled even Microsoft, who referenced `utf8` in their docs, which only works because their input is always ASCII, which makes `utf8.encode()` a no-op.
https://docs.microsoft.com/en-us/rest/api/eventhub/generate-sas-token#nodejs
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the README's copyright-character example and compare it with the Buffer and hashing examples described in the issue. Update the documentation to distinguish output that only looks like UTF-8 from actual UTF-8 bytes, and ensure the example explains the hashing consequence shown in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100