facebookresearch / facebookresearch/AugLy

Other similarities

Open
#73 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
5.1k
Forks
312
PR merge metrics
No merged PRs in 30d

Description

## 🚀 Feature
Replace random word with a phonetically similar one.

Or also replace a random word with the same Part Of Speech or lemma (adjective with adjective or run with ran / running etc...)

## Motivation
I'm training a transformer based model to spell check utterances (like a reversed augly).
Like `Hello r u fin tdy => Hello are you fine today`.

I realized that quite often the spelling errors come from phonetically similar words
exemple (not so good exemple but for the sake of the explanation) : `"I love jeans"` vs `"I love gins"`

Also, augmenting by replacing with sane pos or other inflections of the same lemma would help in the same direction (as better destroying the sentences to train a better spellchecking model)

Having this kind of built-in Augmentation would help building better models.

## Pitch
Having a built-in augmenter that create mistakes not only with levensthein like distances but uses phonetics.
I've done mine using epitran for phonetics and spacy for pos but other frameworks exists.

## Alternatives
Implement my own augmenter (done).
Use only text based distances which cannot find `jean` vs `gin` or `cute` vs `beautiful ` or `run` vs `running` as they are textually too different but often found in chats.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.