Operation request: Remove Diacritics (with replacing letters) / Remove non-latin letters / Latinize
- Dominant language
- JavaScript
- Stars
- 35.8k
- Forks
- 4.1k
- Avg merge
- 2d 26m
- Merged PRs (30d)
- 33
Description
## Summary
Remove diacritics and replace non-latin characters with latin equivalents.
### Example Input
```
ĄĆĘŁŃÓŚŹŻ ąćęłńóśźż æ ø
```
### Example Output
```
ACELNOSZZ acelnoszz ae o
```
## Explanation
There is existing operation `Remove Diacritics`. [Example](https://gchq.github.io/CyberChef/#recipe=Remove_Diacritics()&input=xITEhsSYxYHFg8OTxZrFucW7IMSFxIfEmcWCxYTDs8WbxbrFvCDDpiDDuA).
Example input:
```
ĄĆĘŁŃÓŚŹŻ ąćęłńóśźż æ ø
```
Example output:
```
ACEŁNOSZZ acełnoszz æ ø
```
But you can see that not all letters is replaced. For example in [polish alphabet](https://en.wikipedia.org/wiki/Polish_alphabet#Computer_encoding) letter `ł` isn't replaced to `l`. Another example letters are `æ` and `ø`.
You can think this is a bug, but I found why this is working in that way. [Source code](https://github.com/gchq/CyberChef/blob/master/src/core/operations/RemoveDiacritics.mjs#L35) links to [StackOverflow answer](https://stackoverflow.com/questions/990904/remove-accents-diacritics-in-a-string-in-javascript/37511463#37511463) with implementation. There is discussion and explanation:
> - There's always one char that fails.. `'ąśćńżółźćę'.normalize('NFD').replace(/[\u0300-\u036f]/g, "")` -> `ascnzołzce` (missing match for `ł` ->`l`).
> - `ł` is a letter in itself, not an accented `L`. Therefore it should not be changed to `l`.
So it only removes diacritics, but not removes non-latin letters.
On [Stackoverflow question](https://stackoverflow.com/questions/990904/remove-accents-diacritics-in-a-string-in-javascript/37511463) there are many other answers handling these cases (I think the only way is listing all cases and replacing). Also there is a library latinize (https://github.com/dundalek/latinize) that handled these cases ([source code](https://github.com/dundalek/latinize/blob/master/latinize.js) lists all cases).
### New operation or parameter to existing
I wonder what will be better:
1. New operation (name: `Latinize` or `Remove non-latin letters`).
2. Modify existing operation and add boolean parameter (name `Replace letters` or `Remove non-latin letters` ).
First is more natural because this is another operation. But is causes more and more similar operations.
Contributor guide
Assessment
This issue has not been assessed yet.