Mixed workflow warmup loses model residency on GB10 unified memory

### Feature Idea

I am running ComfyUI as a single service process on an NVIDIA DGX Spark style unified-memory system.

Expected behavior:
After warming multiple workflows in one ComfyUI process, models should remain reusable when the memory budget allows it.

Example:

1. Warm an image workflow
2. Warm a video workflow
3. Warm an audio workflow
4. Run image -> video -> image

The second image run should use the already warmed image models instead of paying the cold-load cost again.

Actual behavior:
Same-workflow repeated runs are fast, but switching to another workflow type causes the previous workflow’s models to lose reusable loaded state.

Observed example:

* First image run: about 120s
* Repeated image run: about 9s
* image -> video -> image: image returns to about 120s again

Similar behavior happens with video -> audio -> video.

Startup flags tested:
--highvram
--gpu-only
--disable-async-offload

These improve same-workflow reuse, but do not make models stay reusable across mixed workflow categories.

Why this matters:
On large unified-memory machines, a single service process may have enough memory to keep several workflows warm. However, ComfyUI currently behaves as if the previous workflow’s model state is no longer reusable after switching workflow categories. This makes mixed image/video/audio API services pay cold-start latency repeatedly.

Possible cause:
The current LoadedModel lifecycle appears to depend on weak references / execution cache ownership. After switching workflows, older ModelPatcher objects may be garbage collected, so even if some memory is still reserved by PyTorch, ComfyUI can no longer reuse the model as a loaded model.

Unified-memory systems also make CUDA free-memory accounting harder, because CUDA, PyTorch, host memory and cgroup limits may report different views of available memory. This can make eviction or partial unload decisions too conservative.

Related issues:
#14276 #13139 #11930 #10896 #11106 #11332

This is related to the issues above, but the focus here is specifically mixed workflow warmup, unified-memory-safe accounting, and explicit resident model semantics for service deployments.

As a temporary workaround, I am running one dedicated ComfyUI instance per workflow type (e.g., one instance solely for image generation, another for video).

While this avoids the cross-workflow cache eviction issue, it requires an API router and wastes resources by duplicating CUDA contexts. It also fails to leverage the global dynamic allocation of Unified Memory systems. A native single-instance solution remains highly necessary.

### Existing Solutions

_No response_

### Other

_No response_

Reproduce the image → video → image and video → audio → video sequences, including the listed startup flags, and inspect the LoadedModel lifecycle and ModelPatcher ownership described in the report. Done means warmed models remain reusable across workflow categories when memory allows, without repeatedly paying the cold-load cost on unified-memory systems.

Written by the indexing model from the issue text.