facebookresearch / facebookresearch/stopes

How to create training data through pipeline

Open
#14 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
313
Forks
47
PR merge metrics
No merged PRs in 30d

Description

I want to train the NLLB model, as instructed by the data [ReadMe](https://github.com/facebookresearch/fairseq/tree/nllb/examples/nllb/data) documentation, I have tried the filtering pipeline and got the output of `populate_data_conf.py` and `compute_length_factors.py`. But I don't know how to run prepare_data pipeline. Especially the three parameters required by `prepare_data.py`, such as the yaml file required by the `--data-config` parameter, etc. Could you provide an example? Thanks a lot.

In addition, what is the relationship between filtering pipeline and prepare_data pipeline? The latter doesn't seem to use the output of the former.

The `compute_length_factors.py` used in the filtering pipeline doesn't seem to be updated as it requires the flores101 dataset instead of the flores200..

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.