pytorch / pytorch/pytorch

Load nVidia libraries in reverse order so that the newest version can take control

Open
#171,092 1 comment 0 reactions 0 assignees View on GitHub
module: binaries module: cuda module: third_party needs reproduction triaged
Dominant language
Python
Stars
103k
Forks
29.5k
PR merge metrics
PR metrics pending

Description

### 🚀 The feature, motivation and pitch

PyTorch tries to be helpful to load all .so libraries in nvidia cudnn directory. But upgrade of nvidia cudnn does not automatically delete old versions, so old and new versions of .so libraries co-exist. PyTorch loads .so libraries in alphabetical order, so old version is loaded first whose version report function would take control in the global symbol table ("First-Come, First-Served" rule of the Linux dynamic linker). Usually, people upgrade packages in order to use the newer version. Loading old version and calling functions in old version is thus not usually intended. The current alphabetical loading convention, however, contradicts this intention. So, I propose abandoning such a self-righteous goodwill of loading all .so nvidia libraries in the directory but only the newest version, or, if PyTorch has to, loading them in reverse alphabetical order so that the newest version can take control in the global symbol table.

This is a specific related problem: https://github.com/NVIDIA/TransformerEngine/issues/2538

Thanks.

### Alternatives

_No response_

### Additional context

_No response_

cc @seemethere @malfet @atalman @tinglvv @nWEIdia @ptrblck @msaroufim @eqy @jerryzh168

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.