Comfy insists on unloading and reloading AnimateDiff Models

Hey there, when going past the context length of 16 frames, Comfy does the following:

[AnimateDiffEvo] - INFO - Loading motion module mm_sdxl_v10_beta.ckpt
[AnimateDiffEvo] - INFO - Using fp16, converting motion module to fp16
E:\StabilityMatrix\Packages\ComfyUI\custom_nodes\ComfyUI_FizzNodes\BatchFuncs.py:128: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
cur_prompt_series[f] = ''
E:\StabilityMatrix\Packages\ComfyUI\custom_nodes\ComfyUI_FizzNodes\BatchFuncs.py:129: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
nxt_prompt_series[f] = ''
Requested to load SDXLClipModel
Loading 1 new model
[AnimateDiffEvo] - INFO - Sliding context window activated - latents passed in (340) greater than context_length 16.
[AnimateDiffEvo] - INFO - Injecting motion module mm_sdxl_v10_beta.ckpt version v1.
Requested to load SDXL
Loading 1 new model
unload clone 1
44%|\u2588\u2588\u2588\u2588\u258c | 11/25 [20:46<26:24, 113.18s/it]

I'm not entirely sure what this is, but it slows down generations a great deal.

The larger issue is, that after doing this, it will be forever stuck in a loop of unloading and reloading this "clone". Even if the workflow doesn't contain any of these models. The only option I had before to fix this, was to uninstall and reinstall Comfy altogether. I, perhaps foolishly, attempted this again, just to run into the same problem. At times, if will just (after several minutes) reload/reinstall said model (forcibly) but then afterwards just get stuck in a loop of "got prompt" and then giving a few second generation time, before repeating "got prompt" verbatim. Again, it will do this on repeat, regardless of me actually asking it to generate anything, and regardless of the actual workflow in front of it. I can generate things in-between this reloading and/or "got prompt" loop, but it obviously slows things down tremendously. Using AnimateDiff, past 16 frames, appears to break Comfy permanently for me.

I appreciate this may or may not be an issue with AnimateDiff (or AnimateDiff evolved) more specifically, but tbh - I have no clue.

[AnimateDiffIssue.json](https://github.com/comfyanonymous/ComfyUI/files/13382262/AnimateDiffIssue.json)

This is the workflow btw.

As a side-note, I use Comfy through Stability Matrix.

Start with the attached AnimateDiffIssue.json and reproduce the path beyond 16 frames that logs “Sliding context window activated.” Inspect the model-loading and clone behavior shown in the ComfyUI logs, alongside custom_nodes/ComfyUI_FizzNodes/BatchFuncs.py:128-129, to determine whether the unload/reload loop comes from ComfyUI or a custom node. Done means the workflow no longer leaves ComfyUI repeatedly unloading and reloading models.

Written by the indexing model from the issue text.