huggingface / huggingface/datatrove
Periodical logging of stats
- Dominant language
- Python
- Stars
- 3.3k
- Forks
- 302
- Avg merge
- 2h 18m
- Merged PRs (30d)
- 2
Description
For long running tasks it would be useful to emit statistics every once in a while, let's say every 60s.
It's frustrating to have to wait for the pipeline to finish just to see that maybe throughput isn't great or something else is the matter. It would be useful to be able to monitor as the data gets processed.
I haven't found anything built in for that. Did I miss something?
Would you be interested in accepting such a MR?
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are named. Start by tracing the pipeline's existing statistics and logging paths, then determine how a 60-second reporting interval would fit; done should mean that long-running processing emits useful statistics while it is still running.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100