huggingface / huggingface/evaluate

Wrong accuracy on the validation set!!!

Open
#500 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

Hi All,

I fine tuned "distilbert-base-uncased" for text classification on a custom dataset. The best accuracy on the validation set was 0.9009 as you can see here: https://huggingface.co/anuragrawal/distilbert-base-uncased-finetuned

I am new to using the evaluate library and wanted to use it for my test set. Before doing that, I thought of doing an accuracy check on the validation set but the accuracy is different than 0.9009. I trained my model for only 2 epochs and the 2nd epoch had the best accuracy. When I use evaluate on my validation set, the accuracy is 0.893326717935996 which surprisingly was the accuracy on the validation set at the end of 1st epoch. Is this an issue with the evaluate library? or am I missing something?

I followed this page for using the evaluate library: https://huggingface.co/docs/evaluate/base_evaluator

Let me know if you need any additional information.

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.