google / google/uncertainty-baselines

Question about batch size and test-set evaluation

Open
#303 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.6k
Forks
224
Avg merge
15h 36m
Merged PRs (30d)
2

Description

Hi there!

I noticed something a little odd while evaluating an ensemble using `baselines/cifar/ensemble.py`: it seems that evaluation is only performed on the test set rounded down to a multiple of the batch size, rather than the full set. I noticed this as the numpy arrays which store the predictions have shape (9984, 10) (that script has an eff. batch size of 64, which divides 9984).

I believe that this might be the case in the other training/eval scripts as well; as I read it, the test iterator is only called for the first `TEST_IMAGES // BATCH_SIZE` batches, leaving a partial batch if the batch size doesn't evenly divide.

Please let me know if I'm mistaken about this. If you find this is accurate, do the reported results need to be reevaluated? If they were run with the current default effective batch size of 512, I believe 272 test examples out of 10000 were missed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.