facebookresearch / facebookresearch/stopes

Prepare new data for NLLB-200

Open
#24 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
313
Forks
47
PR merge metrics
No merged PRs in 30d

Description

Hi, I'm trying to fine tune NLLB-200 model on new bilingual data. So I need to prepare my data using prepare_data pipeline: https://github.com/facebookresearch/stopes/tree/main/stopes/pipelines/prepare_data
there are my configs file:
![image](https://user-images.githubusercontent.com/50990415/217801860-8ad849fe-6208-4d08-abd2-48509555d432.png)

![image](https://user-images.githubusercontent.com/50990415/217801943-afd853a2-94b9-41dd-a870-d4a508380826.png)

![image](https://user-images.githubusercontent.com/50990415/217802077-829996fb-f1f0-424f-ae29-502c7c6325d1.png)

My output directory is the following:
![image](https://user-images.githubusercontent.com/50990415/217802492-cd6bf829-83c7-4e9a-8448-fe347f3c41c6.png)

But I encountered a problem when fine tuning NLLb-200:
**File "/home/admin/khadija/fairseq/slurm_snapshot_code/2023-02-08T14_51_26.242208/fairseq/data/dictionary.py", line 238, in add_from_file
with open(PathManager.get_local_path(f), "r", encoding="utf-8") as fd:
FileNotFoundError: [Errno 2] No such file or directory: '/home/admin/khadija/prepare_data_output/data_bin/shard000/dict.ary_Arab.txt'
srun: error: slurmnode1: tasks 0-2: Exited with exit code 1**

Is Fairseq compatible with the new version of Stopes?
@Mortimerp9 @kauterry @gwenzek Can you help me please?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.