Load nVidia libraries in reverse order so that the newest version can take control
- Dominant language
- Python
- Stars
- 103k
- Forks
- 29.5k
- PR merge metrics
- PR metrics pending
Description
### 🚀 The feature, motivation and pitch
PyTorch tries to be helpful to load all .so libraries in nvidia cudnn directory. But upgrade of nvidia cudnn does not automatically delete old versions, so old and new versions of .so libraries co-exist. PyTorch loads .so libraries in alphabetical order, so old version is loaded first whose version report function would take control in the global symbol table ("First-Come, First-Served" rule of the Linux dynamic linker). Usually, people upgrade packages in order to use the newer version. Loading old version and calling functions in old version is thus not usually intended. The current alphabetical loading convention, however, contradicts this intention. So, I propose abandoning such a self-righteous goodwill of loading all .so nvidia libraries in the directory but only the newest version, or, if PyTorch has to, loading them in reverse alphabetical order so that the newest version can take control in the global symbol table.
This is a specific related problem: https://github.com/NVIDIA/TransformerEngine/issues/2538
Thanks.
### Alternatives
_No response_
### Additional context
_No response_
cc @seemethere @malfet @atalman @tinglvv @nWEIdia @ptrblck @msaroufim @eqy @jerryzh168
Contributor guide
Assessment
This issue has not been assessed yet.