Model tiiuae/falcon-mamba-7b hangs when using nvfuser executor

@naoyam is already working on this.

Since Aug 25, 2025.

*Note*: If you have a model or program that is not supported yet but should be, please use the program coverage template.

## 🐛 Bug

env:
pjnl-20250727
thunder: 5359a400782423fe1b4e8f0798d5cdf3fb580ae4
nvfuser 0.2.28+gitd268897

`python thunder/benchmarks/benchmark_peft.py --model tiiuae/falcon-mamba-7b --trust-remote-code --attn-implementation eager --max-steps 10 --mbs 1 --seq-length 128 --compile thunder --fixed-num-hidden-layers 2`

model hangs with nvfuser executor. But when removing the nvfuser executor the model works.
```
diff --git a/thunder/benchmarks/benchmark_peft.py b/thunder/benchmarks/benchmark_peft.py
index e1cc6ffc..194f6db2 100644
--- a/thunder/benchmarks/benchmark_peft.py
+++ b/thunder/benchmarks/benchmark_peft.py
@@ -269,7 +269,8 @@ def setup_compilation(model, backend: str):
from thunder.dynamo import thunderfx

# TODO get parameters out from thunderfx CompiledObject
- compiled_object = thunderfx(model, transforms=xforms, executors=executors)
+ # ['cudnn', 'sdpa', 'torchcompile_xentropy', 'nvfuser']
+ compiled_object = thunderfx(model, transforms=xforms, executors=[thunder.cudnn_executor, thunder.sdpa_executor, thunder.torchcompile_xentropy_executor])
model = compiled_object._func

return model
```

Log and debugging information when using default executor list
```
root@ac7777eba444:/wayan/lightning-thunder# python thunder/benchmarks/benchmark_peft.py --model tiiuae/falcon-mamba-7b --trust-remote-code --attn-implementation eager --max-steps 10 --mbs 1 --seq-length 128 --compile thunder --fixed-num-hidden-layers 2
25-07-29 07:29:13 INFO Namespace(model='tiiuae/falcon-mamba-7b', max_steps=10, skip_iters=2, mbs=1, grad_acc_steps=1, seq_length=128, var_seq_length=False, compile='thunder', verbose=False, trust_remote_code=True, fixed_num_hidden_layers=2, attn_implementation='eager', gradient_checkpointing=False)
25-07-29 07:29:13 INFO Global batch size: 1 (mbs: 1, grad_acc_steps: 1, world_size: 1)
25-07-29 07:29:13 INFO Loading tokenizer
25-07-29 07:29:13 INFO Loading base model on meta device...
The fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. Falling back to the sequential implementation of Mamba, as use_mambapy is set to False. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. For the mamba.py backend, follow https://github.com/alxndrTL/mamba.py.
25-07-29 07:29:14 INFO Base model loaded on meta device
25-07-29 07:29:14 INFO Configured model for static shapes with sequence length: 128
25-07-29 07:29:14 INFO Gradient checkpointing disabled
25-07-29 07:29:14 INFO Applying LoRA to model
25-07-29 07:29:14 INFO LoRA applied to model
25-07-29 07:29:14 INFO Verifying gradient setup...
25-07-29 07:29:14 INFO Applying compilation: thunder to model
25-07-29 07:29:14 INFO Resetting cache size for torch.dynamo
/usr/local/lib/python3.12/dist-packages/transformer_engine/__init__.py:59: RuntimeWarning: Detected a Jax installation but could not find the shared object file for the Transformer Engine Jax extension library. If this is not intentional, please reinstall Transformer Engine with `pip install transformer_engine[jax]` or build from source with `NVTE_FRAMEWORK=jax`.
warnings.warn(
25-07-29 07:29:14 INFO Disabled gradient checkpointing for Thunder compilation
25-07-29 07:29:14 INFO Thunder used executors: ['cudnn', 'sdpa', 'torchcompile_xentropy', 'nvfuser']
25-07-29 07:29:14 INFO Applying Thunder compilation with 4 executors
25-07-29 07:29:14 INFO Using ThunderFX
25-07-29 07:29:14 INFO Compilation applied to model
25-07-29 07:29:14 INFO Trainable parameters: 1,197,056
25-07-29 07:29:14 INFO Total parameters: 744,502,272
25-07-29 07:29:14 INFO Trainable parameter ratio: 0.16%
Map: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:00<00:00, 145.10 examples/s]
25-07-29 07:29:15 INFO Starting training...
Training: 0%| | 0/10 [00:00, std::equal_to, std::allocator > const&, std::vector > const&, bool, bool, bool, bool) ()
from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#3 0x00007037a588f0ab in nvfuser::IterVisitor::traverseTo(std::vector > const&, bool, bool, bool, bool) ()
from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#4 0x00007037a588fcfd in nvfuser::StmtSort::getStmtsTo(std::vector > const&, bool, bool, bool) ()
from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#5 0x00007037a588fe93 in nvfuser::StmtSort::getExprsTo(std::vector > const&, bool, bool, bool) ()
from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#6 0x00007037a58903fb in nvfuser::StmtSort::getExprs(nvfuser::Fusion const*, bool, bool, bool) () from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#7 0x00007037a569a5f8 in nvfuser::Fusion::exprs() const () from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#8 0x00007037a5b306a9 in nvfuser::scheduler_tools::(anonymous namespace)::getIndexedConsumerToProducerMap(nvfuser::Fusion*, nvfuser::ComputeAtMap const&) ()
from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#9 0x00007037a5b351ff in nvfuser::scheduler_tools::PointwiseDomainMap::findReferenceTensor(long) const ()
from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#10 0x00007037a5af838d in nvfuser::pointwise_utils::getReferenceTensor(nvfuser::Fusion*) () from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#11 0x00007037a5af02fb in nvfuser::hasReferenceTensorView(nvfuser::Fusion*) () from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#12 0x00007037a5af0911 in nvfuser::PointWiseScheduler::canScheduleCompileTime(nvfuser::Fusion*) () from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#13 0x00007037a5b0b444 in nvfuser::Schedule::proposeHeuristics(nvfuser::Fusion*, nvfuser::SchedulerRuntimeInfo&) ()
from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#14 0x00007037a56de17d in nvfuser::SegmentCandidateFinder::segment(std::unique_ptr >, nvfuser::KernelArgumentHolder const&, nvfuser::SchedulerRuntimeInfo&) () from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#15 0x00007037a5a4e401 in nvfuser::FusionKernelRuntime::FusionKernelRuntime(std::unique_ptr >, nvfuser::KernelArgumentHolder const&, nvfuser::serde::FusionKernelRuntime const*, std::optional, long, long, long, bool) ()
from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#16 0x00007037a5a42c78 in nvfuser::FusionExecutorCache::getKernelRuntimeFor(nvfuser::KernelArgumentHolder const&, std::optional) ()
from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#17 0x00007037a5a436ea in nvfuser::FusionExecutorCache::runFusionWithInputs(nvfuser::KernelArgumentHolder, std::optional, std::optional) ()
from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#18 0x00007037a5c56a11 in nvfuser::python_frontend::FusionDefinition::execute(nvfuser::KernelArgumentHolder, std::optional, bool, bool, bool, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector, std::allocator >, std::allocator, std::allocator > > >) const () from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#19 0x00007037a526d3dd in nvfuser::python_frontend::initNvFuserPythonBindings(_object*)::{lambda(nvfuser::python_frontend::FusionDefinition&, pybind11::iterable const&, std::optional, bool, bool, bool, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector, std::allocator >, std::allocator, std::allocator > > >)#1}::operator()(nvfuser::python_frontend::FusionDefinition&, pybind11::iterable const&, std::optional, bool, bool, bool, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector, std::allocator >, std::allocator, std::allocator > > >) const [clone .isra.0] () from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#20 0x00007037a5344b0d in pybind11::cpp_function::initialize, bool, bool, bool, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector, std::allocator >, std::allocator, std::allocator > > >)#1}, std::pair >, std::vector > >, nvfuser::python_frontend::FusionDefinition&, pybind11::iterable const&, std::optional, bool, bool, bool, std::vector for more, q to quit, c to continue without paging--c
11::basic_string, std::allocator >, std::allocator, std::allocator > > >, std::vector, std::allocator >, std::allocator, std::allocator > > >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg, pybind11::kw_only, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::return_value_policy>(nvfuser::python_frontend::initNvFuserPythonBindings(_object*)::{lambda(nvfuser::python_frontend::FusionDefinition&, pybind11::iterable const&, std::optional, bool, bool, bool, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector, std::allocator >, std::allocator, std::allocator > > >)#1}&&, std::pair >, std::vector > > (*)(nvfuser::python_frontend::FusionDefinition&, pybind11::iterable const&, std::optional, bool, bool, bool, std::vector, std::allocator >, std::allocator, std::allocator > > >, std::vector, std::allocator >, std::allocator, std::allocator > > >), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg const&, pybind11::kw_only const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::return_value_policy const&)::{lambda(pybind11::detail::function_call&)#1}::_FUN(pybind11::detail::function_call&) () from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#21 0x00007037a525baa3 in pybind11::cpp_function::dispatcher(_object*, _object*, _object*) () from /opt/pytorch/nvfuser/python/nvfuser/_C.cpython-312-x86_64-linux-gnu.so
#22 0x0000000000581d4f in ?? ()
#23 0x000000000054904e in _PyObject_MakeTpCall ()
#24 0x00000000005d6b2f in _PyEval_EvalFrameDefault ()
#25 0x000000000054a7d2 in _PyObject_Call_Prepend ()
#26 0x00000000005a3148 in ?? ()
#27 0x0000000000548f85 in _PyObject_MakeTpCall ()
#28 0x00000000005d6b2f in _PyEval_EvalFrameDefault ()
#29 0x000000000054cacd in ?? ()
#30 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#31 0x000000000054cacd in ?? ()
#32 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#33 0x000000000054a7d2 in _PyObject_Call_Prepend ()
#34 0x00000000005a3148 in ?? ()
#35 0x000000000054b13c in PyObject_Call ()
#36 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#37 0x000000000054cacd in ?? ()
#38 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#39 0x000000000054cacd in ?? ()
#40 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#41 0x000000000054cacd in ?? ()
#42 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#43 0x000000000054a7d2 in _PyObject_Call_Prepend ()
#44 0x00000000005a3148 in ?? ()
#45 0x000000000054b13c in PyObject_Call ()
#46 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#47 0x000000000054cacd in ?? ()
#48 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#49 0x00007039b24fdd14 in dynamo.custom_eval_frame () from /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so
#50 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#51 0x00007039b24fdc3d in dynamo.custom_eval_frame () from /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so
#52 0x000000000054cacd in ?? ()
#53 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#54 0x00007039b24fdc3d in dynamo.custom_eval_frame () from /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so
#55 0x000000000054cacd in ?? ()
#56 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#57 0x00007039b24fdd14 in dynamo.custom_eval_frame () from /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so
#58 0x000000000054a7d2 in _PyObject_Call_Prepend ()
#59 0x00000000005a3148 in ?? ()
#60 0x0000000000548f85 in _PyObject_MakeTpCall ()
#61 0x00000000005d6b2f in _PyEval_EvalFrameDefault ()
#62 0x00007039b24fcb0e in dynamo_eval_custom_code () from /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so
#63 0x00007039b24fdb3c in dynamo.custom_eval_frame () from /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so
#64 0x000000000054cb94 in ?? ()
#65 0x000000000054b1b9 in PyObject_Call ()
#66 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#67 0x00007039b24fdd14 in dynamo.custom_eval_frame () from /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so
#68 0x000000000054cb94 in ?? ()
#69 0x000000000054b1b9 in PyObject_Call ()
#70 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#71 0x00007039b24fdd14 in dynamo.custom_eval_frame () from /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so
#72 0x000000000054cb94 in ?? ()
#73 0x000000000054b1b9 in PyObject_Call ()
#74 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#75 0x000000000054cb94 in ?? ()
#76 0x000000000054b1b9 in PyObject_Call ()
#77 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#78 0x000000000054cb94 in ?? ()
#79 0x000000000054b1b9 in PyObject_Call ()
#80 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#81 0x000000000054a89a in _PyObject_Call_Prepend ()
#82 0x00000000005a3148 in ?? ()
#83 0x000000000054b13c in PyObject_Call ()
#84 0x00000000005dad16 in _PyEval_EvalFrameDefault ()
#85 0x00000000005d500b in PyEval_EvalCode ()
#86 0x00000000006081e2 in ?? ()
#87 0x00000000006b5033 in ?? ()
#88 0x00000000006b4d9a in _PyRun_SimpleFileObject ()
#89 0x00000000006b4bcf in _PyRun_AnyFileObject ()
#90 0x00000000006bcc35 in Py_RunMain ()
#91 0x00000000006bc71d in Py_BytesMain ()
#92 0x00007039beb1b1ca in __libc_start_call_main (main=main@entry=0x518950, argc=argc@entry=17, argv=argv@entry=0x7ffdf4c504c8) at ../sysdeps/nptl/libc_start_call_main.h:58
#93 0x00007039beb1b28b in __libc_start_main_impl (main=0x518950, argc=17, argv=0x7ffdf4c504c8, init=, fini=, rtld_fini=,
stack_end=0x7ffdf4c504b8) at ../csu/libc-start.c:360
#94 0x00000000006575a5 in _start ()
```

cc @tfogal