import torch crashes or hangs with spindle
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 110
- Forks
- 35
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 2
Description
import torch sometimes works and sometimes crashes randomly with the following error on 128 nodes.
Could you tag if its an already known issue?
I used commits till 92bda828efcef13e3b2f83e0c01b8280665c290c on devel branch
pytorch 2.5
```
kaushikvelusamy@node_name:lustre_dir_location/kaushik> mpiexec --spindle=python-prefix=${CONDA_DIR} --np 24 --ppn 12 --genvall --genv=PYTHONPATH=${CONDA_DIR} --cpu-bind=list:4:9:14:19:20:25:56:61:66:71:74:79 ${APP_LOC}
spindle-bin/bin/spindle --location=/tmp/kaushik/spindle-test/spindle-temp-dir mpiexec --genvall --genv TMPDIR=/tmp/kaushik/spindle-test/spindle-temp-dir --genv=PYTHONPATH=lustre_dir_location/kaushik/spindle/spindle_tests_apps/torch-with-spindle/lus_pip_torch_2.3_env_128/ --np 1536 --ppn 12 --cpu-bind=list:4:9:14:19:20:25:56:61:66:71:74:79 spindlemarker lustre_dir_location/kaushik/spindle/spindle_tests_apps/torch-with-spindle/app_3.sh
Traceback (most recent call last):
File "", line 1, in
Traceback (most recent call last):
File "", line 1, in
File "lustre_dir_location/kaushik/spindle/spindle_tests_apps/torch-with-spindle/lus_pip_torch_2.3_env_2/torch/__init__.py", line 36, in
from .torch_version import __version__ as __version__
File "lustre_dir_location/kaushik/spindle/spindle_tests_apps/torch-with-spindle/lus_pip_torch_2.3_env_2/torch/torch_version.py", line 5, in
Traceback (most recent call last):
File "", line 1, in
File "lustre_dir_location/kaushik/spindle/spindle_tests_apps/torch-with-spindle/lus_pip_torch_2.3_env_2/torch/__init__.py", line 36, in
from .torch_version import __version__ as __version__
File "lustre_dir_location/kaushik/spindle/spindle_tests_apps/torch-with-spindle/lus_pip_torch_2.3_env_2/torch/torch_version.py", line 5, in
from ._vendor.packaging.version import Version, InvalidVersion
ValueError: source code string cannot contain null bytes
node_name.hostmgmt2611.cm.system_name.: rank 6 exited with code 1
node_name.hostmgmt2611.cm.system_name.: rank 1 died from signal 15
kaushikvelusamy@node_name:lustre_dir_location/kaushik>
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the intermittent failure with the provided spindle and mpiexec commands on 128 nodes, then inspect torch/__init__.py and torch/torch_version.py where the traceback occurs. Determine whether spindle is introducing null bytes into the imported source and document whether the behavior is a known issue, with a reliable reproduction or explanation as the outcome.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100