huggingface / huggingface/datatrove
Wrong stats in multi-node local executor
Open
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
Due to lack of node-level communication, the stats at the end of each pipeline step can only aggregate results from the current node, and what's being written to the disk is the status of the last finished worker, rather than the global info
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the multi-node local executor and the code that aggregates pipeline-step stats and writes status to disk. Trace how worker results are collected, then verify that the persisted output represents global node information rather than the last finished worker.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100