awslabs / awslabs/sagemaker-debugger
Loss Tensors Are Saved Twice On AWS Pytorch
Open
- Dominant language
- Python
- Stars
- 165
- Forks
- 82
- PR merge metrics
- No merged PRs in 30d
Description
loss functional values are saved twice for each step with AWS Pytorch.
This happens because functional losses are saved by default by the post_hook_for_loss_functional fn in AWS Pytorch.
```
...
for _ in range(n_steps):
optimizer.zero_grad()
outputs = net(inputs)
loss = F.cross_entropy(outputs, labels)
hook.record_tensor_value("nll_loss", tensor_value=loss)
loss.backward()
optimizer.step()
...
```
The `post_hook_for_loss_functional` is called by `F.cross_entropy(outputs, labels)`
Contributor guide
Assessment
This issue has not been assessed yet.