FusedLayerNorm: incorrect results when input device doesn't match default device
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Description
Repro:
import torch
from apex.normalization.fused_layer_norm import FusedLayerNormFunction
def layer_norm(device, dim=8):
torch.cuda.manual_seed_all(0)
input = torch.rand(1, 1, dim, device=device)
normalized_shape = torch.Size([dim])
eps = 1e-05
return FusedLayerNormFunction.apply(input, normalized_shape, eps)
print(layer_norm('cuda:0'))
# tensor([[[-0.6874, -0.2992, -1.9222, 1.0983, 1.1173, 0.6251, -0.6347,
# 0.7028]]], device='cuda:0')
print(layer_norm('cuda:1'))
# tensor([[[0., 0., 0., 0., 0., 0., 0., 0.]]], device='cuda:1')
with torch.cuda.device('cuda:1'):
print(layer_norm('cuda'))
# tensor([[[-0.6874, -0.2992, -1.9222, 1.0983, 1.1173, 0.6251, -0.6347,
# 0.7028]]], device='cuda:1')
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the FusedLayerNormFunction entry point and run the provided reproduction on cuda:0, cuda:1, and the current CUDA device. Compare the outputs for inputs on non-default devices; done means the results are consistent regardless of the input device, with coverage for this multi-device case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100