facebookresearch / facebookresearch/stopes
How to create training data through pipeline
- Dominant language
- Python
- Stars
- 313
- Forks
- 47
- PR merge metrics
- No merged PRs in 30d
Description
I want to train the NLLB model, as instructed by the data [ReadMe](https://github.com/facebookresearch/fairseq/tree/nllb/examples/nllb/data) documentation, I have tried the filtering pipeline and got the output of `populate_data_conf.py` and `compute_length_factors.py`. But I don't know how to run prepare_data pipeline. Especially the three parameters required by `prepare_data.py`, such as the yaml file required by the `--data-config` parameter, etc. Could you provide an example? Thanks a lot.
In addition, what is the relationship between filtering pipeline and prepare_data pipeline? The latter doesn't seem to use the output of the former.
The `compute_length_factors.py` used in the filtering pipeline doesn't seem to be updated as it requires the flores101 dataset instead of the flores200..
Contributor guide
Assessment
This issue has not been assessed yet.