Comfy-Org / Comfy-Org/ComfyUI

Bad File Descriptor Errors During Model Loading

Open
#7,124 1 comment 2 reactions 0 assignees View on GitHub
Potential Bug
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 10h
Merged PRs (30d)
153

Description

### Expected Behavior

## Issue Description
When loading large diffusion models in ComfyUI, I'm more often than not encountering `[Errno 9] Bad file descriptor` errors. The issue appears to be a race condition related to file handling during model loading where file descriptors are accessed after they were closed. The error frequency increases with model size - smaller models sometimes work, but larger models like Wan2.1 (14B I2V) consistently fail to load.

Although I've had it over unknown amount of versions, I hadn't reported this yet assuming it must be some file-system issue on my end's configuration, however we see others on Windows 10 local machines are now experiencing exactly the same (python 3.12.7, torch 2.6.0).

## Environment (Virtual Machine hosts)
- **ComfyUI version:** 0.3.24
- **Python version:** 3.10.12 (GCC 11.4.0)
- **PyTorch version:** 2.6.0+cu126
- **CUDA version:** 12.6
- **CuDNN version:** 90501
- **GPU:** NVIDIA A100-SXM4-80GB (using cudaMallocAsync)
- **CPU:** AMD EPYC 7513 32-Core Processor
- **OS:** Linux 5.15.0-134-generic (x86_64) (Ubuntu 22.04 LTS)

## Error Message
```
!!! Exception during processing !!! [Errno 9] Bad file descriptor
```

## What I've Tried
1. Increased soft ulimit to 4096
2. Increased hard ulimit to 65536
3. Increased `fs.inotify.max_user_watches` to 350838
4. Tried implementing a suggested fix:
```python
import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')
```
But this resulted in:
```
RuntimeError: config[i] == get()->name() INTERNAL ASSERT FAILED at "/pytorch/c10/cuda/CUDAAllocatorConfig.cpp":250, please report a bug to PyTorch. Allocator backend parsed at runtime != allocator backend parsed at load time
```

### Actual Behavior

.

### Steps to Reproduce

- Basic ComfyUI setup on Linux or Windows
- Issue occurs when loading large diffusion models (consistently with Wan2.1 and randomly with Hunyuan)
- Problem persists across different hardware configurations (tested on H100/A100/RTX 4090)
- Error occurs randomly but more frequently with larger model sizes
- Models have been verified to not be corrupted during download
- Parameters:
```py
python3 /home/user/ComfyUI/main.py \
--listen \
--port 8888 \
--multi-user \
--extra-model-paths-config /home/user/extra_model_paths.yaml \
--disable-auto-launch \
--preview-method latent2rgb \
--enable-cors-header "*"
>> "$LOGFILE" 2>&1 &
```

### Debug Logs

```powershell
Adding extra search path diffusion_models /home/user/models/diffusion_models
Adding extra search path LLM /home/user/models/LLM
Adding extra search path vae /home/user/models/vae
Adding extra search path loras /home/user/models/loras
Adding extra search path clip /home/user/models/clip
Adding extra search path clip_vision /home/user/models/clip_vision
[START] Security scan
[DONE] Security scan
## ComfyUI-Manager: installing dependencies done.
** ComfyUI startup time: 2025-03-07 21:23:23.456
** Platform: Linux
** Python version: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0]
** Python executable: /home/user/venv/bin/python3
** ComfyUI Path: /home/user/ComfyUI
** ComfyUI Base Folder Path: /home/user/ComfyUI
** User directory: /home/user/ComfyUI/user
** ComfyUI-Manager config path: /home/user/ComfyUI/user/default/ComfyUI-Manager/config.ini
** Log path: /home/user/ComfyUI/user/comfyui.log

Prestartup times for custom nodes:
4.3 seconds: /home/user/ComfyUI/custom_nodes/ComfyUI-Manager

Checkpoint files will always be loaded safely.
Total VRAM 81342 MB, total RAM 45149 MB
pytorch version: 2.6.0+cu126
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA A100-SXM4-80GB : cudaMallocAsync
Using pytorch attention
ComfyUI version: 0.3.24
[Prompt Server] web root: /home/user/venv/lib/python3.10/site-packages/comfyui_frontend_package/static
### Loading: ComfyUI-Manager (V3.27.8)
[ComfyUI-Manager] network_mode: public
### ComfyUI Version: v0.3.24-8-gd60fe0af | Released on '2025-03-07'

Import times for custom nodes:
0.0 seconds: /home/user/ComfyUI/custom_nodes/ComfyUI-Manager

Starting server

To see the GUI go to: http://0.0.0.0:8888
To see the GUI go to: http://[::]:8888
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
[ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
FETCH ComfyRegistry Data: 5/36
FETCH ComfyRegistry Data: 10/36
FETCH ComfyRegistry Data: 15/36
FETCH ComfyRegistry Data: 20/36
FETCH ComfyRegistry Data: 25/36
FETCH ComfyRegistry Data: 30/36
FETCH ComfyRegistry Data: 35/36
FETCH ComfyRegistry Data [DONE]
[ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
nightly_channel:
https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/remote
FETCH DATA from: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json [DONE]
[ComfyUI-Manager] All startup tasks have been completed.
got prompt
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
Requested to load CLIPVisionModelProjection
loaded completely 79698.675 1208.09814453125 True
Requested to load WanTEModel
loaded completely 9.5367431640625e+25 10835.4765625 True
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cuda:0, dtype: torch.float16
Requested to load WanVAE
loaded completely 65358.1728477478 242.02829551696777 True
model weight dtype torch.float16, manual cast: None
model_type FLOW
Requested to load WAN21
!!! Exception during processing !!! [Errno 9] Bad file descriptor
```

### Other

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.