TensorAdapter silently fails to load in PyTorch, significantly hurting performance
- Dominant language
- Python
- Stars
- 14.3k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
## 🐛 Bug
As of 0.8, if the `libtensoradapter_pytorch` fails to load, it is not reported to users, and they will see performance drop by up to 2x and in increase in GPU memory (and in some cases get confusing out-of-memory errors from pytorch where it is only using a subset of the memory).
This is a bad user experience, and opens DGL up to inaccurate (and unfavorable) comparisons.
If a user is using DGL on a GPU, they should get some warning that they are running in a degraded state. I know our previous warning lead to some confusion (#3377), but ignoring it for GPU runs defeats many of optimization efforts.
Since there is no warning, users have no reason to run with `DGL_LOG_DEBUG=1` set. Even if the user does set the environment variable, the current message is:
```
>>> import dgl
2022-05-11 16:46:15,425 base.py:142 DEBUG p:MainProcess t:MainThread: Memory optimization with PyTorch is not enabled.
2022-05-11 16:46:15,426 __init__.py:47 DEBUG p:MainProcess t:MainThread: Using backend: pytorch
```
Which doesn't tell users the impact of 'Memory optimization' being disabled, nor how they might fix it.
I think it would be good to have the warning on by default including instructions on its impact, how to fix it, and how to disable. We could silence the warning if a user's system doesn't have any cuda capable devices (or at least if they didn't build with CUDA support).
If we consider it normal behavior to run without tensoradapter, I think we need to really need to improve our ndarray allocation and memory pool implementation.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.