2017, ACL, Data Augmentation for Low-Resource Neural Machine Translation

**Paper**
Data Augmentation for Low-Resource Neural Machine Translation

**Introduction**
This research focuses on the challenges faced by low-resource languages in the neural machine translation (NMT) domain. NMT, as introduced by Bahdanau et al. (2015), Sutskever et al. (2014), and Cho et al. (2014), follows a sequence-to-sequence architecture. In this architecture, the encoder processes the source language and creates a representation of it, while the decoder—using the same structure (hidden LSTM states and attention mechanisms)—generates the target language. Training such a network requires various words to appear in diverse contexts.

**Main Problem**
The quality of NMT heavily depends on the size of the dataset. To reliably estimate the model’s parameters, many sentence pairs with words in varied contexts is necessary. However, data annotation is time-consuming. Therefore, data augmentation is a viable alternative.

**Illustrative Example**
English: I had been told that you would [not/voluntarily] be speaking today.
German: Mir wurde signalisiert, Sie würden heute [nicht/freiwillig] sprechen.

**Input:**
A sentence in the source language.

**Output:**
A sentence in the target language that is grammatically correct and fluent (They have used BLEU for this purpose) but not necessarily identical in meaning.

**Motivation**
The paper proposes simple data augmentation methods (inspired by those used in computer vision) to improve neural machine translation. The method introduces a weak notion of label preservation, focusing on low-frequency words. In brief, they synthetically provide contexts that are created to provide a foundation for the model to generate rare words in various contexts during translation.
Related Works and Their Gaps
Sennrich et al. (2016a) proposed a method to back-translate sentences from monolingual data and augment the bitext with the resulting pseudo-parallel corpora.
They assert that their proposed method has a better performance.

**Contribution of This Paper**
Compared to paraphrasing, their method introduces more information to the model. They claim their method outperforms back-translation.

**Proposed Methods**
Not included

**Experiments**
Model: A 4-layer attention-based encoder-decoder model by Luong et al. (2015), with a 2-layer LSTM for forward and backward propagation.
Dataset: WMT’15

**Implementation**
Not mentioned

**Gaps in This Work**
This article addresses neural machine translation and the challenges of handling low-resource languages. However, it has some limitations:
The method is limited to rare words. Its practicality depends on whether it can be applied to real-world data, such as social media platforms like Twitter and Instagram.
Since rare words are less frequently used, they do not capture the complexity of common words, which may have various meanings in different contexts (or are used as slang on social media).
Is the method applicable across various language families? The approach may yield better results for languages within the same family compared to languages from different families.