facebookresearch / facebookresearch/silk

jax/jaxlib error occurs in training

Open
#71 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Gnuplot
Stars
679
Forks
66
PR merge metrics
No merged PRs in 30d

Description

Jax cannot find GPU devices, but Pytorch (torch1.11 + cu102) can.

nvcc-V 10.2
jax 0.4.13
jaxlib 0.4.13

I installed Jax using the following command:
`
pip install jax==0.4.13 -f https://storage.googleapis.com/jax-releases/jax_releases.html
pip install jaxlib[cu102]==0.4.13 -f https://storage.googleapis.com/jax-releases/jax_releases.html
`
The training error happened as follows:
`
jaxlib.xla_extension.XlaRuntimeError: INVALID_ARGUMENT: DLPack tensor is on GPU, but no GPU backend was provided.
2024-04-13 16:21:57.832 | ERROR | silk.cli:main:116 - run failed, `*.log` file might be found in : var/silk-cli/run/training/2024-04-13/16-21-46
`

Could you please give me some suggestions or instructions?
Thanks for your time!

Haolin

Contributor guide

Open the contributing guide

Research direction

Start at silk.cli:main:116 and inspect the referenced var/silk-cli/run/training/2024-04-13/16-21-46 log for the full training failure. Reproduce the reported JAX 0.4.13 and jaxlib 0.4.13 installation with CUDA 10.2, compare the detected backends with PyTorch, and consider the issue addressed when GPU training runs without the DLPack XlaRuntimeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.