facebookresearch / facebookresearch/stopes
Using stopes with an unseen language
Open
- Dominant language
- Python
- Stars
- 313
- Forks
- 47
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I'm trying to clean and preprocess bitext for finetuning NLLB on a new unseen language. The source language is a part of laser3, the target language is not included. Will it work if I replace laser3 with BPE encoder pre-trained on my target language?
Thank you!
`python -m stopes.pipelines.bitext.global_mining_pipeline src_lang=fuv tgt_lang=zul demo_dir=.../stopes-repo/demo +preset=demo output_dir=. embed_text=laser3`
Contributor guide
Assessment
This issue has not been assessed yet.