flash_attn_2_cuda.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEa

### Custom Node Testing

- [ ] I have tried disabling custom nodes and the issue persists (see [how to disable custom nodes](https://docs.comfy.org/troubleshooting/custom-node-issues#step-1%3A-test-with-all-custom-nodes-disabled) if you need help)

### Expected Behavior

No error and nodes should work

### Actual Behavior

Some nodes (also base comfyui install nodes) dont work with the error `flash_attn_2_cuda.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEa`

### Steps to Reproduce

No idea

### Debug Logs

```powershell
Traceback (most recent call last):
File "/config/05-comfy-ui/ComfyUI/nodes.py", line 2149, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 999, in exec_module
File "", line 488, in _call_with_frames_removed
File "/config/05-comfy-ui/ComfyUI/custom_nodes/comfyui_essentials/__init__.py", line 2, in
from .image import IMAGE_CLASS_MAPPINGS, IMAGE_NAME_MAPPINGS
File "/config/05-comfy-ui/ComfyUI/custom_nodes/comfyui_essentials/image.py", line 8, in
import kornia
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/__init__.py", line 25, in
from . import (
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/augmentation/__init__.py", line 20, in
from kornia.augmentation._2d import (
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/augmentation/_2d/__init__.py", line 19, in
from kornia.augmentation._2d.intensity import *
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/augmentation/_2d/intensity/__init__.py", line 47, in
from kornia.augmentation._2d.intensity.plasma import RandomPlasmaBrightness, RandomPlasmaContrast, RandomPlasmaShadow
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/augmentation/_2d/intensity/plasma.py", line 22, in
from kornia.contrib import diamond_square
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/contrib/__init__.py", line 32, in
from .image_stitching import ImageStitcher
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/contrib/image_stitching.py", line 24, in
from kornia.feature import LocalFeatureMatcher, LoFTR
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/feature/__init__.py", line 24, in
from .integrated import (
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/feature/integrated.py", line 34, in
from .lightglue import LightGlue
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/feature/lightglue.py", line 48, in
from flash_attn.modules.mha import FlashCrossAttention
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/flash_attn/__init__.py", line 3, in
from flash_attn.flash_attn_interface import (
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/flash_attn/flash_attn_interface.py", line 15, in
import flash_attn_2_cuda as flash_attn_gpu
ImportError: /config/05-comfy-ui/env/lib/python3.12/site-packages/flash_attn_2_cuda.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEa

Cannot import /config/05-comfy-ui/ComfyUI/custom_nodes/comfyui_essentials module for custom nodes: /config/05-comfy-ui/env/lib/python3.12/site-packages/flash_attn_2_cuda.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEa
Using pytorch attention
Traceback (most recent call last):
File "/config/05-comfy-ui/ComfyUI/nodes.py", line 2149, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 999, in exec_module
File "", line 488, in _call_with_frames_removed
File "/config/05-comfy-ui/ComfyUI/custom_nodes/RES4LYF/__init__.py", line 7, in
from . import images
File "/config/05-comfy-ui/ComfyUI/custom_nodes/RES4LYF/images.py", line 11, in
import kornia
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/__init__.py", line 25, in
from . import (
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/augmentation/__init__.py", line 20, in
from kornia.augmentation._2d import (
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/augmentation/_2d/__init__.py", line 19, in
from kornia.augmentation._2d.intensity import *
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/augmentation/_2d/intensity/__init__.py", line 47, in
from kornia.augmentation._2d.intensity.plasma import RandomPlasmaBrightness, RandomPlasmaContrast, RandomPlasmaShadow
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/augmentation/_2d/intensity/plasma.py", line 22, in
from kornia.contrib import diamond_square
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/contrib/__init__.py", line 32, in
from .image_stitching import ImageStitcher
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/contrib/image_stitching.py", line 24, in
from kornia.feature import LocalFeatureMatcher, LoFTR
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/feature/__init__.py", line 24, in
from .integrated import (
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/feature/integrated.py", line 34, in
from .lightglue import LightGlue
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/feature/lightglue.py", line 48, in
from flash_attn.modules.mha import FlashCrossAttention
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/flash_attn/__init__.py", line 3, in
from flash_attn.flash_attn_interface import (
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/flash_attn/flash_attn_interface.py", line 15, in
import flash_attn_2_cuda as flash_attn_gpu
ImportError: /config/05-comfy-ui/env/lib/python3.12/site-packages/flash_attn_2_cuda.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEa

Cannot import /config/05-comfy-ui/ComfyUI/custom_nodes/RES4LYF module for custom nodes: /config/05-comfy-ui/env/lib/python3.12/site-packages/flash_attn_2_cuda.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEa
Traceback (most recent call last):
File "/config/05-comfy-ui/ComfyUI/nodes.py", line 2149, in load_custom_node
module_spec.loader.exec_module(module)
File "", line 999, in exec_module
File "", line 488, in _call_with_frames_removed
File "/config/05-comfy-ui/ComfyUI/custom_nodes/comfyui-vrgamedevgirl/__init__.py", line 1, in
from .nodes import NODE_CLASS_MAPPINGS as NODE_CLASS_MAPPINGS_2, NODE_DISPLAY_NAME_MAPPINGS as NODE_DISPLAY_NAME_MAPPINGS_2
File "/config/05-comfy-ui/ComfyUI/custom_nodes/comfyui-vrgamedevgirl/nodes.py", line 6, in
import kornia
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/__init__.py", line 25, in
from . import (
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/augmentation/__init__.py", line 20, in
from kornia.augmentation._2d import (
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/augmentation/_2d/__init__.py", line 19, in
from kornia.augmentation._2d.intensity import *
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/augmentation/_2d/intensity/__init__.py", line 47, in
from kornia.augmentation._2d.intensity.plasma import RandomPlasmaBrightness, RandomPlasmaContrast, RandomPlasmaShadow
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/augmentation/_2d/intensity/plasma.py", line 22, in
from kornia.contrib import diamond_square
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/contrib/__init__.py", line 32, in
from .image_stitching import ImageStitcher
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/contrib/image_stitching.py", line 24, in
from kornia.feature import LocalFeatureMatcher, LoFTR
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/feature/__init__.py", line 24, in
from .integrated import (
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/feature/integrated.py", line 34, in
from .lightglue import LightGlue
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/kornia/feature/lightglue.py", line 48, in
from flash_attn.modules.mha import FlashCrossAttention
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/flash_attn/__init__.py", line 3, in
from flash_attn.flash_attn_interface import (
File "/config/05-comfy-ui/env/lib/python3.12/site-packages/flash_attn/flash_attn_interface.py", line 15, in
import flash_attn_2_cuda as flash_attn_gpu
ImportError: /config/05-comfy-ui/env/lib/python3.12/site-packages/flash_attn_2_cuda.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEa
```

### Other

I am using the container provided here https://github.com/grokuku/stable-diffusion under unraid. I have tried reinstalling comfyui but couldnt get it to work either.

This is only an issue with some nodes, not all. Most work, but some throw this error. Currently lost how this should be solved.