gchq / gchq/CyberChef

Operation request: Remove Diacritics (with replacing letters) / Remove non-latin letters / Latinize

Open
#1,251 2 comments 0 reactions 0 assignees View on GitHub
operation
Dominant language
JavaScript
Stars
35.8k
Forks
4.1k
Avg merge
2d 26m
Merged PRs (30d)
33

Description

## Summary

Remove diacritics and replace non-latin characters with latin equivalents.

### Example Input

```
ĄĆĘŁŃÓŚŹŻ ąćęłńóśźż æ ø
```

### Example Output

```
ACELNOSZZ acelnoszz ae o
```

## Explanation

There is existing operation `Remove Diacritics`. [Example](https://gchq.github.io/CyberChef/#recipe=Remove_Diacritics()&input=xITEhsSYxYHFg8OTxZrFucW7IMSFxIfEmcWCxYTDs8WbxbrFvCDDpiDDuA).

Example input:
```
ĄĆĘŁŃÓŚŹŻ ąćęłńóśźż æ ø
```

Example output:
```
ACEŁNOSZZ acełnoszz æ ø
```

But you can see that not all letters is replaced. For example in [polish alphabet](https://en.wikipedia.org/wiki/Polish_alphabet#Computer_encoding) letter `ł` isn't replaced to `l`. Another example letters are `æ` and `ø`.

You can think this is a bug, but I found why this is working in that way. [Source code](https://github.com/gchq/CyberChef/blob/master/src/core/operations/RemoveDiacritics.mjs#L35) links to [StackOverflow answer](https://stackoverflow.com/questions/990904/remove-accents-diacritics-in-a-string-in-javascript/37511463#37511463) with implementation. There is discussion and explanation:

> - There's always one char that fails.. `'ąśćńżółźćę'.normalize('NFD').replace(/[\u0300-\u036f]/g, "")` -> `ascnzołzce` (missing match for `ł` ->`l`).
> - `ł` is a letter in itself, not an accented `L`. Therefore it should not be changed to `l`.

So it only removes diacritics, but not removes non-latin letters.

On [Stackoverflow question](https://stackoverflow.com/questions/990904/remove-accents-diacritics-in-a-string-in-javascript/37511463) there are many other answers handling these cases (I think the only way is listing all cases and replacing). Also there is a library latinize (https://github.com/dundalek/latinize) that handled these cases ([source code](https://github.com/dundalek/latinize/blob/master/latinize.js) lists all cases).

### New operation or parameter to existing

I wonder what will be better:
1. New operation (name: `Latinize` or `Remove non-latin letters`).
2. Modify existing operation and add boolean parameter (name `Replace letters` or `Remove non-latin letters` ).

First is more natural because this is another operation. But is causes more and more similar operations.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.