OOM eventually when using create_graph=True with BatchL2Grad
- Dominant language
- Python
- Stars
- 617
- Forks
- 57
- PR merge metrics
- No merged PRs in 30d
Description
I was trying to use my second-order optimizer [ESGD-M](https://github.com/crowsonkb/esgd) with BatchL2Grad in order to collect information on within-batch gradient variance to estimate stochastic noise (think OpenAI's gradient noise scale paper), and I kept OOMing after maybe six epochs of MNIST training. ESGD-M does a Hessian-vector product internally (not using Backpack stuff, just autograd) so it needs the user to specify create_graph=True. I assume when I use it with Backpack, something is leaking references to past computational graphs, normally these graphs are garbage collected without issue.
Thank you,
Katherine Crowson
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.