Deterministic implementation of SparseMatrix softmax
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
Hi, I ran the example of [Graph Transformer](https://docs.dgl.ai/notebooks/sparse/graph_transformer.html?highlight=graph%20transformer) and met some problems in reproducibility.
First, I set all random seed as follows:
```
def setup_seed(seed):
torch.manual_seed(seed)
random.seed(seed)
torch.cuda.manual_seed_all(seed)
torch.cuda.manual_seed(seed)
```
The results still (accuracy) varies, for example , acc=0.72, acc=0.85 ...
So I try to set `torch.use_deterministic_algorithms(True)` to true. But I met another problem:
```
File "E:\YuAnHuang\kevislin\second_proj\scALGCN\model.py", line 184, in forward
h = layer(A, h)
File "E:\Software\Anaconda\envs\kevislin\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "E:\YuAnHuang\kevislin\second_proj\scALGCN\model.py", line 108, in forward
h = self.MHA(A, h)
File "E:\Software\Anaconda\envs\kevislin\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "E:\YuAnHuang\kevislin\second_proj\scALGCN\model.py", line 73, in forward
attn = attn.softmax() # (sparse) [N, N, nh]
File "E:\Software\Anaconda\envs\kevislin\lib\site-packages\dgl\sparse\softmax.py", line 71, in softmax
torch.ops.dgl_sparse.softmax(input.c_sparse_matrix, dim)
File "E:\Software\Anaconda\envs\kevislin\lib\site-packages\torch\_ops.py", line 502, in __call__
return self._op(*args, **kwargs or {})
RuntimeError: scatter_reduce_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation, or you can use the 'warn_only=True' option, if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.
```
It seems like that the implementation of softmax of SparseMatrix is not deterministic, I wonder if there is a way to solve this problem?
CUDA: 11.8
PyTorch: 2.0
Python: 3.9
DGL: 1.1.2_cu118
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.