facebookresearch / facebookresearch/SlowFast
Huge memory footprint in evaluation
Open
- Dominant language
- Python
- Stars
- 7.4k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
During the evaluation epoch, outputs of models and labels are added to a list and thus remain in the gpu memory, see [here](https://github.com/facebookresearch/SlowFast/blob/64abcc90ccfdcbb11cf91d6e525bed60e92a8796/tools/train_net.py#L255).
This will give rise to `RuntimeError: CUDA out of memory.` when the validation set is of large scale.
Is there any way to compute the metrics on the fly?
Contributor guide
Assessment
This issue has not been assessed yet.