deepmodeling / deepmodeling/Uni-Mol
往logging_output写日志时导致内存泄露问题
Open
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 181
- PR merge metrics
- No merged PRs in 30d
Description
在模型训练过程发现内存(非显存)泄露问题,现象为随着训练轮次增加内存一直往上涨,最后定位到原因是在损失类里如下语句导致
logging_output = {
"loss": loss,
}
改为
logging_output = {
"loss": loss.data
}
问题:为何少加了.data,会导致内存泄露呢?
不加.data,reduce_metrics收到数据为
outputs [{'sample_size': 1, 'loss': tensor(5.6797, device='cuda:0', dtype=torch.float16,
grad_fn=)}]
加上.data为
outputs [{'sample_size': 1, 'loss': tensor(5.6797, device='cuda:0', dtype=torch.float16,)}]
看起来是不加.data,多了个grad_fn(激活函数?)导致,为何?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.