google / google/uncertainty-baselines
Question about batch size and test-set evaluation
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 224
- Avg merge
- 15h 36m
- Merged PRs (30d)
- 2
Description
Hi there!
I noticed something a little odd while evaluating an ensemble using `baselines/cifar/ensemble.py`: it seems that evaluation is only performed on the test set rounded down to a multiple of the batch size, rather than the full set. I noticed this as the numpy arrays which store the predictions have shape (9984, 10) (that script has an eff. batch size of 64, which divides 9984).
I believe that this might be the case in the other training/eval scripts as well; as I read it, the test iterator is only called for the first `TEST_IMAGES // BATCH_SIZE` batches, leaving a partial batch if the batch size doesn't evenly divide.
Please let me know if I'm mistaken about this. If you find this is accurate, do the reported results need to be reevaluated? If they were run with the current default effective batch size of 512, I believe 272 test examples out of 10000 were missed.
Contributor guide
Assessment
This issue has not been assessed yet.