Lightning-AI / Lightning-AI/lightning-thunder
custom_op executor AssertionError when running different tests together
@crcrpar is already working on this.
Since Nov 3, 2025.
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 121
- PR merge metrics
- No merged PRs in 30d
Description
With `pjnl-20251103`
Repro:
```
$ pytest thunder/tests/test_recipes.py thunder/tests/test_torch_library_custom_op.py thunder/tests/test_torch_library_custom_op_with_lists.py
```
Log
```
___________________________________________________________________________________________________________________________ test_plugins_basics ____________________________________________________________________________________________________________________________
@pytest.mark.skipif(not nvfuser_available(), reason="nvFuser is not available")
def test_plugins_basics():
model = torch.nn.Sequential(torch.nn.Linear(2048, 4096), torch.nn.ReLU(), torch.nn.Linear(4096, 64))
from thunder import compile_data as get_compile_data
thunder_model = thunder.compile(model)
x = torch.randn(64, 2048)
_ = thunder_model(x)
cd = get_compile_data(thunder_model)
assert cd is not None
for ex in get_expected_executors():
> assert ex.name in [el.name for el in cd.executors_list]
E AssertionError: assert 'custom_op' in ['__ad_hoc_executor_129167309031136', '__ad_hoc_executor_129167309030176', 'nvfuser', 'torch', 'python']
E + where 'custom_op' = thunder.extend.OperatorExecutor('custom_op').name
thunder/tests/test_recipes.py:200: AssertionError
_________________________________________________________________________________________________________________________ test_plugins_composition _________________________________________________________________________________________________________________________
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x757a1bb59100>
@pytest.mark.skipif(not nvfuser_available(), reason="nvFuser is not available")
@pytest.mark.skipif(IS_WINDOWS, reason="libuv error with PT build on windows")
def test_plugins_composition(monkeypatch):
model = torch.nn.Sequential(torch.nn.Linear(2048, 4096), torch.nn.ReLU(), torch.nn.Linear(4096, 64))
monkeypatch.setenv("LOCAL_RANK", "0")
with patch("thunder.jit") as mock_jit:
_ = thunder.compile(model, plugins="fp8")
call_args = mock_jit.call_args
assert "transformer_engine_v1" in [el.name for el in call_args.kwargs["executors"]]
for ex in get_expected_executors():
> assert ex.name in [el.name for el in call_args.kwargs["executors"]]
E AssertionError: assert 'custom_op' in ['__ad_hoc_executor_129167308652272', 'transformer_engine_v1', 'nvfuser']
E + where 'custom_op' = thunder.extend.OperatorExecutor('custom_op').name
thunder/tests/test_recipes.py:216: AssertionError
========================================================================================================================= short test summary info ==========================================================================================================================
FAILED thunder/tests/test_recipes.py::test_plugins_basics - AssertionError: assert 'custom_op' in ['__ad_hoc_executor_129167309031136', '__ad_hoc_executor_129167309030176', 'nvfuser', 'torch', 'python']
FAILED thunder/tests/test_recipes.py::test_plugins_composition - AssertionError: assert 'custom_op' in ['__ad_hoc_executor_129167308652272', 'transformer_engine_v1', 'nvfuser']
=============================================================================================================== 2 failed, 17 passed, 113 warnings in 46.96s ================================================================================================================
```
NOTE: The test runs fine when run independently - `pytest thunder/tests/test_recipes.py`
cc: @crcrpar
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.