FGRibreau / FGRibreau/node-unidecode

Multi-pass required to correctly unidecode

Open
#16 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
151
Forks
24
PR merge metrics
No merged PRs in 30d

Description

For some cases it needs several passes to completely convert to ascii. Take this example:

var unidecode = require('unidecode')
var s = 'Rocío Martín-Valero'; // there is a hidden - appearing after both Ã's if you paste in console!
console.log(unidecode(s)) // prints RocÃo MartÃn-Valero (removes that hidden -), but still not ascii
console.log(unidecode(unidecode(s))) // 2 passes to print RocAo MartAn-Valero

Here is the hexdump of the above string:

00000000 52 6f 63 c3 83 c2 ad 6f 20 4d 61 72 74 c3 83 c2 |Roc....o Mart...|
00000010 ad 6e 2d 56 61 6c 65 72 6f 0a |.n-Valero.|
0000001a

So it seems it can't convert the 2 sequences `c3 83` and `c2 ad` that are back to back.

---
Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/41747210-multi-pass-required-to-correctly-unidecode?utm_campaign=plugin&utm_content=tracker%2F294213&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F294213&utm_medium=issues&utm_source=github).

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied `unidecode(s)` reproduction and tracing the package entry point used by `require('unidecode')`. Check how the adjacent byte sequences are handled and add regression coverage for the reported string; done means one call produces the expected ASCII output.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.