Lightning-AI / Lightning-AI/lightning-thunder

test_transformer_engine_executor.py is not runnable in the PyTorch build without distributed support

Open
#1,350 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ci / tests
Dominant language
Python
Stars
1.5k
Forks
121
PR merge metrics
No merged PRs in 30d

Description

## 🐛 Bug

1. Modify `is_available()` function to return False in `torch/distributed/__init__.py`
2. Modify `is_available()` function to return False in `torch/distributed/rpc/__init__.py`
3. Run `pytest thunder/tests/test_transformer_engine_executor.py --collect-only`
```py
=============================================================================================== ERRORS ===============================================================================================
_________________________________________________________________ ERROR collecting thunder/tests/test_transformer_engine_executor.py _________________________________________________________________
ImportError while importing test module '/opt/pytorch/lightning-thunder/thunder/tests/test_transformer_engine_executor.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.12/importlib/__init__.py:90: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
thunder/tests/test_transformer_engine_executor.py:13: in
import transformer_engine.pytorch as te
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/__init__.py:64: in
from transformer_engine.pytorch.module import LayerNormLinear
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/module/__init__.py:6: in
from .layernorm_linear import LayerNormLinear
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/module/layernorm_linear.py:15: in
from .base import (
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/module/base.py:28: in
from ..distributed import (
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/distributed.py:16: in
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
/usr/local/lib/python3.12/dist-packages/torch/distributed/fsdp/__init__.py:1: in
from ._flat_param import FlatParameter as FlatParameter
/usr/local/lib/python3.12/dist-packages/torch/distributed/fsdp/_flat_param.py:45: in
from torch.testing._internal.distributed.fake_pg import FakeProcessGroup
/usr/local/lib/python3.12/dist-packages/torch/testing/_internal/distributed/fake_pg.py:5: in
from torch._C._distributed_c10d import (
E ModuleNotFoundError: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package
_________________________________________________________________ ERROR collecting thunder/tests/test_transformer_engine_executor.py _________________________________________________________________
ImportError while importing test module '/opt/pytorch/lightning-thunder/thunder/tests/test_transformer_engine_executor.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.12/importlib/__init__.py:90: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
thunder/tests/test_transformer_engine_executor.py:13: in
import transformer_engine.pytorch as te
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/__init__.py:64: in
from transformer_engine.pytorch.module import LayerNormLinear
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/module/__init__.py:6: in
from .layernorm_linear import LayerNormLinear
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/module/layernorm_linear.py:15: in
from .base import (
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/module/base.py:28: in
from ..distributed import (
/usr/local/lib/python3.12/dist-packages/transformer_engine/pytorch/distributed.py:16: in
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
/usr/local/lib/python3.12/dist-packages/torch/distributed/fsdp/__init__.py:1: in
from ._flat_param import FlatParameter as FlatParameter
/usr/local/lib/python3.12/dist-packages/torch/distributed/fsdp/_flat_param.py:45: in
from torch.testing._internal.distributed.fake_pg import FakeProcessGroup
/usr/local/lib/python3.12/dist-packages/torch/testing/_internal/distributed/fake_pg.py:5: in
from torch._C._distributed_c10d import (
E ModuleNotFoundError: No module named 'torch._C._distributed_c10d'; 'torch._C' is not a package
====================================================================================== short test summary info =======================================================================================
ERROR thunder/tests/test_transformer_engine_executor.py
ERROR thunder/tests/test_transformer_engine_executor.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================================================================== no tests collected, 2 errors in 1.46s ================================================================================
```

cc @borda

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running `pytest thunder/tests/test_transformer_engine_executor.py --collect-only` and inspect the import at line 13 of `thunder/tests/test_transformer_engine_executor.py`. Reproduce the failure with distributed support unavailable, then trace the imports through `torch/distributed/__init__.py` and `torch/distributed/rpc/__init__.py`; done means collection completes without the reported import errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.