facebookresearch / facebookresearch/AugLy
Other similarities
- Dominant language
- Python
- Stars
- 5.1k
- Forks
- 312
- PR merge metrics
- No merged PRs in 30d
Description
## 🚀 Feature
Replace random word with a phonetically similar one.
Or also replace a random word with the same Part Of Speech or lemma (adjective with adjective or run with ran / running etc...)
## Motivation
I'm training a transformer based model to spell check utterances (like a reversed augly).
Like `Hello r u fin tdy => Hello are you fine today`.
I realized that quite often the spelling errors come from phonetically similar words
exemple (not so good exemple but for the sake of the explanation) : `"I love jeans"` vs `"I love gins"`
Also, augmenting by replacing with sane pos or other inflections of the same lemma would help in the same direction (as better destroying the sentences to train a better spellchecking model)
Having this kind of built-in Augmentation would help building better models.
## Pitch
Having a built-in augmenter that create mistakes not only with levensthein like distances but uses phonetics.
I've done mine using epitran for phonetics and spacy for pos but other frameworks exists.
## Alternatives
Implement my own augmenter (done).
Use only text based distances which cannot find `jean` vs `gin` or `cute` vs `beautiful ` or `run` vs `running` as they are textually too different but often found in chats.
Contributor guide
Assessment
This issue has not been assessed yet.