deepmodeling / deepmodeling/Uni-Mol

往logging_output写日志时导致内存泄露问题

Open
#173 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
181
PR merge metrics
No merged PRs in 30d

Description

在模型训练过程发现内存(非显存)泄露问题,现象为随着训练轮次增加内存一直往上涨,最后定位到原因是在损失类里如下语句导致
logging_output = {
"loss": loss,
}
改为
logging_output = {
"loss": loss.data
}
问题:为何少加了.data,会导致内存泄露呢?

不加.data,reduce_metrics收到数据为
outputs [{'sample_size': 1, 'loss': tensor(5.6797, device='cuda:0', dtype=torch.float16,
grad_fn=)}]
加上.data为
outputs [{'sample_size': 1, 'loss': tensor(5.6797, device='cuda:0', dtype=torch.float16,)}]
看起来是不加.data,多了个grad_fn(激活函数?)导致,为何?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.