facebookresearch / facebookresearch/large_concept_model
Data preprocessing with multi-gpus
Open
- Dominant language
- Python
- Stars
- 2.4k
- Forks
- 213
- PR merge metrics
- No merged PRs in 30d
Description
Hi LCM teams, thanks for your great work, I am trying to use your prepare_wikipedia script to deal with my own data. Right now, I have a 1 node with 8 gpus, but I found that this pipeline is super slow, and it seems that only this is only using one gpu. Could u help me to figure out how to use multi-gpus and prallel, with this script?
Contributor guide
Assessment
This issue has not been assessed yet.