huggingface / huggingface/datatrove
In-file parallelism
Open
enhancement
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
Current parallel strategy assign different files in a directory to different workers.
There are many situations where this may incur load unbalancing, for example, when the input files are irregular in size or the input is one single giant file.
Is it possible to implement the functionality of in-file parallelism? For each file, assign different lines to different workers
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.