Large model library causes very slow startup

I am experiencing longer and longer startup times in ComfyUI as the number of models in the extra paths grow. Tested on a completely fresh install with only the extra paths changed (no custom nodes installed). With a few hundred models startup is already much slower than without. I have tens of thousands of models in my full library and it doesn't seem feasible to add them all to the paths at this rate.

I'm not sure what part of ComfyUI is causing this issue, but it would be much appreciated if something could be done so that large model libraries are possible without crippling load times.

No file, test, or entry point is named. Reproduce startup on a fresh install with extra paths containing a few hundred and then tens of thousands of models, and identify what scales poorly. Done means large model libraries no longer make startup impractically slow.

Written by the indexing model from the issue text.