facebookresearch / facebookresearch/silk
jax/jaxlib error occurs in training
- Dominant language
- Gnuplot
- Stars
- 679
- Forks
- 66
- PR merge metrics
- No merged PRs in 30d
Description
Jax cannot find GPU devices, but Pytorch (torch1.11 + cu102) can.
nvcc-V 10.2
jax 0.4.13
jaxlib 0.4.13
I installed Jax using the following command:
`
pip install jax==0.4.13 -f https://storage.googleapis.com/jax-releases/jax_releases.html
pip install jaxlib[cu102]==0.4.13 -f https://storage.googleapis.com/jax-releases/jax_releases.html
`
The training error happened as follows:
`
jaxlib.xla_extension.XlaRuntimeError: INVALID_ARGUMENT: DLPack tensor is on GPU, but no GPU backend was provided.
2024-04-13 16:21:57.832 | ERROR | silk.cli:main:116 - run failed, `*.log` file might be found in : var/silk-cli/run/training/2024-04-13/16-21-46
`
Could you please give me some suggestions or instructions?
Thanks for your time!
Haolin
Contributor guide
Research direction
Start at silk.cli:main:116 and inspect the referenced var/silk-cli/run/training/2024-04-13/16-21-46 log for the full training failure. Reproduce the reported JAX 0.4.13 and jaxlib 0.4.13 installation with CUDA 10.2, compare the detected backends with PyTorch, and consider the issue addressed when GPU training runs without the DLPack XlaRuntimeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100