Use reinforcement learning to improve quality of NMT models for low resource languages

We currently use supervised fine tuning on base NMT models (NLLB, MADLAD) to build a model for a particular minority language pair. LLMs use reinforcement learning on top of SFT to further improve the model. Reinforcement Learning from Human Feedback (RLHF) is used on most modern LLMs.

Direct Preference Optimization (DPO) would be the easiest way forward. Preference data are triples: source, preferred translation, dispreferred translation. This could be collected by asking a translation team to select their preferred translation from multiple translations. In practical terms, it would be difficult to collect this data from translation teams. It would be preferrable to find some way to generate synthetic preference data.

The paper, CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation, is an example of how DPO can be applied to NMT without human feedback. Most of the RL techniques use reference-free quality estimation (QE) models, such as COMET-QE, as a reward model. Improving Machine Translation with Human Feedback: An Exploration of Quality Estimation as a Reward Model is another paper that uses QE as a reward model (uses RAFT instead of DPO).

Unfortunately, QE models, such as COMET, only work on a limited number of languages, so it can't be applied to minority languages. RL without QE can be achieved in a couple of ways. Dispreferred translations can be synthetically generated by noising reference translations. Another option is to use Minimum Bayes-Risk (MBR) to rank translations based on consensus scoring. The paper, Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding, lays out an approach for using MBR decoding on a set of source sentences to create preference data.

No files, tests, or entry points are identified. Start by reviewing the existing supervised fine-tuning pipeline for NLLB and MADLAD, then determine how synthetic preference data using DPO, noised references, or MBR would be evaluated; done requires a selected approach and evidence of improved low-resource translation quality.