huggingface / huggingface/datatrove
Some Tasks Remain Incomplete in local_executor
Open
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
I'm working on collecting Korean-language fineweb (kofineweb) and currently working `v0.4.4`
While processing a new snapshot, I noticed an issue that has been recurring. When running tasks in `local_executor`, some tasks remain incomplete. Usually, around 1 to 5 tasks are left unfinished in `total_task`.
Has the cause of this issue been identified? Previously, when I encountered this, I would just discard the stuck tasks and move on to the filtering process. Let me know if there are any updates or recommended fixes.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.