error when running rf

When i run the rf.
the command is: rf3 fold inputs='tests/data/5vht_from_json.json' ckpt_path=/data2/wmq/.foundry/checkpoints/rf3_foundry_01_24_latest_remapped.ckpt
the output is below:
Traceback (most recent call last):
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/cuequivariance_torch/primitives/triangle.py", line 236, in triangle_multiplicative_update
from cuequivariance_ops_torch import triangle_multiplicative_update as f
File "/data2/wmq/.conda/envs/foundry/lib/python3.12/site-packages/cuequivariance_ops_torch/init.py", line 16, in
from cuequivariance_ops_torch.segmented_transpose import (
File "/data2/wmq/.conda/envs/foundry/lib/python3.12/site-packages/cuequivariance_ops_torch/segmented_transpose.py", line 13, in
import cuequivariance_ops_torch._ext as ops
File "/data2/wmq/.conda/envs/foundry/lib/python3.12/site-packages/cuequivariance_ops_torch/_ext/init.py", line 12, in
from .cuequivariance_ops_torch_ext import * # light-weight wrapper
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: /home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/cuequivariance_ops/lib/libcue_ops.so: undefined symbol: cublasGemmGroupedBatchedEx, version libcublas.so.12

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/wmq/.conda/envs/foundry/bin/rf3", line 7, in
sys.exit(app())
^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/typer/main.py", line 336, in call
raise e
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/typer/main.py", line 319, in call
return get_command(self)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/click/core.py", line 1485, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/typer/core.py", line 814, in main
return _main(
^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/typer/core.py", line 190, in _main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/click/core.py", line 1873, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/click/core.py", line 1269, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/click/core.py", line 824, in invoke
return callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/typer/main.py", line 706, in wrapper
return callback(**use_params)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/cli.py", line 69, in fold
run_inference(cfg)
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/hydra/main.py", line 83, in decorated_main
return task_function(cfg_passthrough)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/inference.py", line 63, in run_inference
inference_engine.run(**run_params)
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/inference_engines/rf3.py", line 546, in run
valid_step_outs = self.trainer.validation_step(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/trainers/rf3.py", line 504, in validation_step
network_output = model.forward(
^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/lightning/fabric/wrappers.py", line 136, in forward
output = self._forward_module(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/foundry/training/EMA.py", line 67, in forward
return self.shadow(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/model/RF3.py", line 407, in forward
first_recycle_outputs = next(recycling_output_generator)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/model/RF3.py", line 259, in trunk_forward_with_recycling
recycling_inputs = self.recycle(**recycling_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/model/RF3.py", line 307, in recycle
S_I, Z_II = self.recycler(
^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/model/RF3_structure.py", line 296, in forward
Z_II = Z_II + self.template_embedder(f, Z_II)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/model/layers/pairformer_layers.py", line 775, in forward
embedded_templates = embed_templates_like_rf3(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/foundry/training/checkpoint.py", line 59, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/model/layers/pairformer_layers.py", line 769, in embed_templates_like_rf3
_, v_II = block(None, v_II)
^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/foundry/training/checkpoint.py", line 59, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/model/layers/pairformer_layers.py", line 388, in forward
self.maybe_make_batched(self.tri_mul_outgoing)(Z_II)
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/model/layers/layer_utils.py", line 107, in _wrap
o = f(arg)
^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/model/layers/attention.py", line 239, in forward
return self._forward_cuequivariance(pair)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/rf3/model/layers/attention.py", line 298, in _forward_cuequivariance
output = cuet.triangle_multiplicative_update(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/wmq/.conda/envs/foundry/lib/python3.12/site-packages/cuequivariance_torch/primitives/triangle.py", line 238, in triangle_multiplicative_update
raise ImportError(
ImportError: Error importing triangle_multiplicative_update from cuequivariance_ops_torch.

I tried to fix it by AI and the env is as below:

_libgcc_mutex 0.1 main https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
_openmp_mutex 5.1 1_gnu https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
absl-py 2.3.1 pypi_0 pypi
aiohappyeyeballs 2.6.1 pypi_0 pypi
aiohttp 3.13.3 pypi_0 pypi
aiosignal 1.4.0 pypi_0 pypi
annotated-types 0.7.0 pypi_0 pypi
antlr4-python3-runtime 4.9.3 pypi_0 pypi
assertpy 1.1 pypi_0 pypi
asttokens 3.0.1 pypi_0 pypi
atomworks 2.2.0 pypi_0 pypi
attrs 25.4.0 pypi_0 pypi
beartype 0.22.9 pypi_0 pypi
biotite 1.4.0 pypi_0 pypi
biotraj 1.2.2 pypi_0 pypi
bzip2 1.0.8 h5eee18b_6 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ca-certificates 2025.12.2 h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
certifi 2026.1.4 pypi_0 pypi
charset-normalizer 3.4.4 pypi_0 pypi
click 8.3.1 pypi_0 pypi
comm 0.2.3 pypi_0 pypi
cuda-cudart 12.4.127 0 nvidia
cuda-nvrtc 12.4.127 0 nvidia
cuequivariance 0.8.1 pypi_0 pypi
cuequivariance-ops-cu12 0.8.1 pypi_0 pypi
cuequivariance-ops-torch-cu12 0.8.1 pypi_0 pypi
cuequivariance-torch 0.8.1 pypi_0 pypi
cython 3.2.4 pypi_0 pypi
cytoolz 0.12.3 pypi_0 pypi
debugpy 1.8.19 pypi_0 pypi
decorator 5.2.1 pypi_0 pypi
dm-tree 0.1.9 pypi_0 pypi
einops 0.8.1 pypi_0 pypi
einx 0.3.0 pypi_0 pypi
environs 11.2.1 pypi_0 pypi
executing 2.2.1 pypi_0 pypi
expat 2.7.3 h7354ed3_4 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
filelock 3.20.3 pypi_0 pypi
frozendict 2.4.7 pypi_0 pypi
frozenlist 1.8.0 pypi_0 pypi
fsspec 2026.1.0 pypi_0 pypi
gitdb 4.0.12 pypi_0 pypi
gitpython 3.1.46 pypi_0 pypi
hydra-core 1.3.2 pypi_0 pypi
hydride 1.2.3 pypi_0 pypi
idna 3.11 pypi_0 pypi
ipykernel 7.1.0 pypi_0 pypi
ipython 9.9.0 pypi_0 pypi
ipython-pygments-lexers 1.1.1 pypi_0 pypi
jaxtyping 0.3.5 pypi_0 pypi
jedi 0.19.2 pypi_0 pypi
jinja2 3.1.6 pypi_0 pypi
jupyter-client 8.8.0 pypi_0 pypi
jupyter-core 5.9.1 pypi_0 pypi
ld_impl_linux-64 2.44 h153f514_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libexpat 2.7.3 h7354ed3_4 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libffi 3.4.4 h6a678d5_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libgcc 15.2.0 h69a1729_7 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libgcc-ng 15.2.0 h166f726_7 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libgomp 15.2.0 h4751f2c_7 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libnsl 2.0.0 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libstdcxx 15.2.0 h39759b7_7 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libstdcxx-ng 15.2.0 hc03a8fd_7 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libuuid 1.41.5 h5eee18b_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libxcb 1.17.0 h9b100fa_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libzlib 1.3.1 hb25bd0a_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
lightning 2.6.0 pypi_0 pypi
lightning-utilities 0.15.2 pypi_0 pypi
loralib 0.1.2 pypi_0 pypi
markdown-it-py 4.0.0 pypi_0 pypi
markupsafe 3.0.3 pypi_0 pypi
marshmallow 4.2.0 pypi_0 pypi
matplotlib-inline 0.2.1 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
msgpack 1.1.2 pypi_0 pypi
multidict 6.7.0 pypi_0 pypi
ncurses 6.5 h7934f7d_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
nest-asyncio 1.6.0 pypi_0 pypi
networkx 3.6.1 pypi_0 pypi
numpy 2.4.1 pypi_0 pypi
nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi
nvidia-cufile-cu12 1.11.1.6 pypi_0 pypi
nvidia-curand-cu12 10.3.5.147 pypi_0 pypi
nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi
nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi
nvidia-cusparselt-cu12 0.6.2 pypi_0 pypi
nvidia-ml-py 13.590.44 pypi_0 pypi
nvidia-nccl-cu12 2.21.5 pypi_0 pypi
nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
nvidia-nvtx-cu12 12.4.127 pypi_0 pypi
omegaconf 2.3.0 pypi_0 pypi
openssl 3.0.18 hd6dcaed_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
opt-einsum 3.4.0 pypi_0 pypi
packaging 25.0 pypi_0 pypi
pandas 2.3.3 pypi_0 pypi
parso 0.8.5 pypi_0 pypi
pexpect 4.9.0 pypi_0 pypi
pillow 12.1.0 pypi_0 pypi
pip 25.3 pyhc872135_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
platformdirs 4.5.1 pypi_0 pypi
prompt-toolkit 3.0.52 pypi_0 pypi
propcache 0.4.1 pypi_0 pypi
protobuf 6.33.3 pypi_0 pypi
psutil 7.2.1 pypi_0 pypi
pthread-stubs 0.3 h0ce48e5_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.3 pypi_0 pypi
py3dmol 2.5.3 pypi_0 pypi
pyarrow 17.0.0 pypi_0 pypi
pydantic 2.12.5 pypi_0 pypi
pydantic-core 2.41.5 pypi_0 pypi
pygments 2.19.2 pypi_0 pypi
pymol-remote 1.1.0 pypi_0 pypi
python 3.12.12 hd17a9e1_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
python-dateutil 2.9.0.post0 pypi_0 pypi
python-dotenv 1.2.1 pypi_0 pypi
pytorch-lightning 2.6.0 pypi_0 pypi
pytz 2025.2 pypi_0 pypi
pyyaml 6.0.3 pypi_0 pypi
pyzmq 27.1.0 pypi_0 pypi
rc-foundry 0.1.9 pypi_0 pypi
rdkit 2025.3.6 pypi_0 pypi
readline 8.3 hc2a1206_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
requests 2.32.5 pypi_0 pypi
rich 14.2.0 pypi_0 pypi
rootutils 1.0.7 pypi_0 pypi
scipy 1.17.0 pypi_0 pypi
sentry-sdk 2.49.0 pypi_0 pypi
setuptools 80.10.2 pypi_0 pypi
shellingham 1.5.4 pypi_0 pypi
six 1.17.0 pypi_0 pypi
smmap 5.0.2 pypi_0 pypi
sqlite 3.51.1 he0a8d7e_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
stack-data 0.6.3 pypi_0 pypi
sympy 1.13.1 pypi_0 pypi
tk 8.6.15 h54e0aa7_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
toolz 1.1.0 pypi_0 pypi
torch 2.6.0+cu124 pypi_0 pypi
torchaudio 2.6.0+cu124 pypi_0 pypi
torchmetrics 1.8.2 pypi_0 pypi
torchvision 0.21.0+cu124 pypi_0 pypi
tornado 6.5.4 pypi_0 pypi
tqdm 4.67.1 pypi_0 pypi
traitlets 5.14.3 pypi_0 pypi
triton 3.3.0 pypi_0 pypi
typer 0.21.1 pypi_0 pypi
typing-extensions 4.15.0 pypi_0 pypi
typing-inspection 0.4.2 pypi_0 pypi
tzdata 2025.3 pypi_0 pypi
urllib3 2.6.3 pypi_0 pypi
wadler-lindig 0.1.7 pypi_0 pypi
wandb 0.23.1 pypi_0 pypi
wcwidth 0.2.14 pypi_0 pypi
wheel 0.45.1 py312h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
wrapt 2.0.1 pypi_0 pypi
xorg-libx11 1.8.12 h9b100fa_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
xorg-libxau 1.0.12 h9b100fa_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
xorg-libxdmcp 1.1.5 h9b100fa_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
xorg-xorgproto 2024.1 h5eee18b_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
xz 5.6.4 h5eee18b_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
yarl 1.22.0 pypi_0 pypi
zlib 1.3.1 hb25bd0a_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
zstandard 0.25.0 pypi_0 pypi

Start with the reported rf3 fold command and the cuequivariance import failure, especially cuequivariance_ops_torch and its libcue_ops.so dependency. Compare the listed environment versions with the rf3 entry points in inference.py and the model path in attention.py; done means the command runs past triangle_multiplicative_update without the undefined-symbol error.