huggingface / huggingface/datatrove

Wrong stats files at the end when the job is relaunch

Open
#360 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.3k
Forks
302
Avg merge
2h 18m
Merged PRs (30d)
2

Description

Hi
after an error, a fail or anything that may interrupt the executor, the stats calculated at the end of the pipeline doesn't takes in account the stats of already accomplished job

I think it will be better if the final stats are the aggregated stats of all the tasks not only the one that just run this time

Thanks for your work, it's a pleasure to work with this tool

Best Regards

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.