huggingface / huggingface/evaluate
Wrong accuracy on the validation set!!!
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
Hi All,
I fine tuned "distilbert-base-uncased" for text classification on a custom dataset. The best accuracy on the validation set was 0.9009 as you can see here: https://huggingface.co/anuragrawal/distilbert-base-uncased-finetuned
I am new to using the evaluate library and wanted to use it for my test set. Before doing that, I thought of doing an accuracy check on the validation set but the accuracy is different than 0.9009. I trained my model for only 2 epochs and the 2nd epoch had the best accuracy. When I use evaluate on my validation set, the accuracy is 0.893326717935996 which surprisingly was the accuracy on the validation set at the end of 1st epoch. Is this an issue with the evaluate library? or am I missing something?
I followed this page for using the evaluate library: https://huggingface.co/docs/evaluate/base_evaluator
Let me know if you need any additional information.
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.