huggingface / huggingface/datatrove
feat request: thread executor
Open
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
Current local executor uses multiprocessing as worker manager, which might be inefficient in io intensive tasks, in which case multi-threading might be a better choice.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.