Comfy-Org / Comfy-Org/ComfyUI

Model reloads for every API request, adding additional 15sec to each inference request. Issue with indexing.

Open
#6,898 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 6h
Merged PRs (30d)
155

Description

> This is a critical bug, but probably a quick fix for the devs.
>
> To get a little more objective, in API mode it's taking ~13-20 seconds for each API call, or ~3.0 seconds through the UI, for the same request. This is completely because of model reloading.
>
>
> In case it helps, a sample of logs:
>
> ```
> Requested to load FluxClipModel_
> got prompt
> loaded partially 7709.3625 7709.36181640625 0
> Requested to load Flux
> loaded partially 7709.3625 7708.8067626953125 239
> 100%|█████| 6/6 [00:02<00:00, 1.25s/it]
> Requested to load AutoencodingEngine
> loaded completely 392.293359375 159.87335777282715 True
> Prompt executed in 20.23 seconds
> Requested to load FluxClipModel_
> loaded partially 7709.3625 7709.36181640625 0
> Requested to load Flux
> loaded partially 7709.3625 7708.8067626953125 239
> 100%|█████| 6/6 [00:07<00:00, 1.26s/it]
> Requested to load AutoencodingEngine
> loaded completely 392.293359375 159.87335777282715 True
> Prompt executed in 13.32 seconds
> got prompt
> Requested to load FluxClipModel_
> loaded partially 7709.3625 7709.36181640625 0
> Requested to load Flux
> loaded partially 7709.3625 7708.8067626953125 239
> ...
> ```
>
> On a whim, I loaded the api workflow in the UI to see if that could somehow force the API model to remain loaded, but alas it does not.
>
> So it appears that `comfy.model_management.load_models_gpu` is getting called each time, and if loaded through the API interface, it appears there's an indexing mismatch, so this is probably a quick fix:
>
> ```python
> for x in models:
> loaded_model = LoadedModel(x)
> try:
> loaded_model_index = current_loaded_models.index(loaded_model)
> except Exception as e:
> loaded_model_index = None
> logging.info(f'current_loaded_models.index fail: {e}') # <<<<<<<<<<<<<<<<<<<<<<
>
> if loaded_model_index is not None:
> loaded = current_loaded_models[loaded_model_index]
> loaded.currently_used = True
> models_to_load.append(loaded)
> else:
> if hasattr(x, "model"):
> logging.info(f"Requested to load {x.model.__class__.__name__}")
> models_to_load.append(loaded_model)
>
> ```
>
> Indeed, logging the `except` branch shows it's failing each time to find the API-loaded model:
>
> ```
> current_loaded_models.index fail: is not in list
> current_loaded_models.index fail: is not in list
> current_loaded_models.index fail: is not in list
> current_loaded_models.index fail: is not in list
> ...
> ```
>
> And note, in UI-only mode, this only happens the first time the model is loaded, and successfully finds the loaded models thereafter.

_Originally posted by @freckletonj in [#2503](https://github.com/comfyanonymous/ComfyUI/issues/2503#issuecomment-2666556599)_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.