[Bug]: "Forge load real model" time is very, very slow (may not respect settings to keep checkpoints loaded)

### Checklist

- [x] The issue exists after disabling all extensions
- [x] The issue exists on a clean installation of webui
- [ ] The issue is caused by an extension, but I believe it is caused by a bug in the webui
- [x] The issue exists in the current version of the webui
- [x] The issue has not been reported before recently
- [ ] The issue has been reported before but has not been fixed yet

### What happened?

The forge load times for a checkpoint are very long, much longer than AUTO1111. I have my settings to keep two checkpoints loaded at all times (one on device and one in system RAM). I have one checkpoint for the main, plus one refiner running. For every single run, it seems the checkpoints get loaded back instead of being pulled from the device VRAM or system RAM.

There is a very large delay to load the refiner, and adding LORA or other models (like Adetailer) slows the system even more. For reference, the current Forge takes between 1m48s to 3m46s to do a 1024x1024x40 step image (with refiner kicking in at 75% and two LORA running).

As a test, I removed the refiner and LORA and the time ranged from 31-33 seconds, which is great, but as soon as the refiner is turned on, the time jumps back up. Placing the refiner back resulted in even slower times in excess of six minutes. All tests were done with no other program running, other than one extra window in Firefox

This problem did not occur about 10 days ago in Forge, an image as described above took about 46-48 seconds. For reference, the current AUTO1111 will consistently output an image with the same settings in ~46 seconds. Both programs are set up with identical parameter and webui settings.

### Steps to reproduce the problem

Launch program
Ensure refiner is set
Run a prompt
Watch the program delay a long time to load/unload the checpoints

### What should have happened?

Output time with refiner should be similar to AUTO1111 at ~46 seconds, and not vary as it does

### What browsers do you use to access the UI ?

Mozilla/Firefox
### Sysinfo

{
"Platform": "Windows-10-10.0.19045-SP0",
"Python": "3.10.6",
"Version": "f0.0.12-latest-155-gd81e353d",
"Commit": "d81e353d8928147bbd973068d0efbb2802affe0f",
"Script path": "C:\\Applications\\StableDiffusion\\Forge",
"Data path": "C:\\Applications\\StableDiffusion\\Forge",
"Extensions dir": "C:\\Applications\\StableDiffusion\\Forge\\extensions",
"Checksum": "cf3a5ac8ca38ba02bafb4d9ac0c663eb3611cf6c676493bc8ae9bbcb30b31dce",
"Commandline": [
"launch.py",
"--xformers",
"--autolaunch",
"--port",
"23743",
"--gradio-auth",
""
],
"Torch env info": {
"torch_version": "2.1.2+cu121",
"is_debug_build": "False",
"cuda_compiled_version": "12.1",
"gcc_version": null,
"clang_version": null,
"cmake_version": "version 3.27.5",
"os": "Microsoft Windows 10 Pro",
"libc_version": "N/A",
"python_version": "3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] (64-bit runtime)",
"python_platform": "Windows-10-10.0.19045-SP0",
"is_cuda_available": "True",
"cuda_runtime_version": null,
"cuda_module_loading": "LAZY",
"nvidia_driver_version": "546.01",
"nvidia_gpu_models": "GPU 0: NVIDIA GeForce RTX 3060",
"cudnn_version": null,
"pip_version": "pip3",
"pip_packages": [
"numpy==1.26.2",
"open-clip-torch==2.20.0",
"pytorch-lightning==1.9.4",
"torch==2.1.2+cu121",
"torchdiffeq==0.2.3",
"torchmetrics==1.3.1",
"torchsde==0.2.6",
"torchvision==0.16.2+cu121"
],
"conda_packages": null,
"hip_compiled_version": "N/A",
"hip_runtime_version": "N/A",
"miopen_runtime_version": "N/A",
"caching_allocator_config": "",
"is_xnnpack_available": "True",
"cpu_info": [
"Architecture=9",
"CurrentClockSpeed=3400",
"DeviceID=CPU0",
"Family=198",
"L2CacheSize=8192",
"L2CacheSpeed=",
"Manufacturer=GenuineIntel",
"MaxClockSpeed=3400",
"Name=13th Gen Intel(R) Core(TM) i7-13700K",
"ProcessorType=3",
"Revision="
]
},
"Exceptions": [],
"CPU": {
"model": "Intel64 Family 6 Model 183 Stepping 1, GenuineIntel",
"count logical": 24,
"count physical": 16
},
"RAM": {
"total": "48GB",
"used": "26GB",
"free": "21GB"
},
"Extensions": [],
"Inactive extensions": [],
"Environment": {
"COMMANDLINE_ARGS": " --xformers --autolaunch --port 23743 --gradio-auth xx:xx",
"GRADIO_ANALYTICS_ENABLED": "False"
},
"Config": {
"samples_save": true,
"samples_format": "png",
"samples_filename_pattern": "[datetime<%Y-%m-%d %H-%M-%S >] [prompt_no_styles]",
"save_images_add_number": false,
"grid_save": false,
"grid_format": "png",
"grid_extended_filename": false,
"grid_only_if_multiple": true,
"grid_prevent_empty_spots": true,
"grid_zip_filename_pattern": "",
"n_rows": -1,
"font": "",
"grid_text_active_color": "#000000",
"grid_text_inactive_color": "#999999",
"grid_background_color": "#ffffff",
"enable_pnginfo": true,
"save_txt": true,
"save_images_before_face_restoration": false,
"save_images_before_highres_fix": false,
"save_images_before_color_correction": false,
"save_mask": false,
"save_mask_composite": false,
"jpeg_quality": 80,
"webp_lossless": false,
"export_for_4chan": true,
"img_downscale_threshold": 8.0,
"target_side_length": 4000,
"img_max_size_mp": 200,
"use_original_name_batch": true,
"use_upscaler_name_as_suffix": false,
"save_selected_only": true,
"save_init_img": false,
"temp_dir": "",
"clean_temp_dir_at_start": false,
"save_incomplete_images": false,
"outdir_samples": "x",
"outdir_txt2img_samples": "x",
"outdir_img2img_samples": "",
"outdir_extras_samples": "",
"outdir_grids": "",
"outdir_txt2img_grids": "outputs/txt2img-grids",
"outdir_img2img_grids": "outputs/img2img-grids",
"outdir_save": "",
"outdir_init_images": "outputs/init-images",
"save_to_dirs": true,
"grid_save_to_dirs": true,
"use_save_to_dirs_for_ui": false,
"directories_filename_pattern": "[datetime<%Y>]\\[datetime<%m - %b>]\\[datetime<%Y-%m-%d>]",
"directories_max_prompt_words": 10,
"ESRGAN_tile": 192,
"ESRGAN_tile_overlap": 8,
"realesrgan_enabled_models": [
"R-ESRGAN 4x+",
"R-ESRGAN 4x+ Anime6B"
],
"upscaler_for_img2img": "ESRGAN_4x",
"face_restoration": false,
"face_restoration_model": "CodeFormer",
"code_former_weight": 0.29,
"face_restoration_unload": true,
"auto_launch_browser": "Local",
"show_warnings": false,
"show_gradio_deprecation_warnings": true,
"memmon_poll_rate": 0,
"samples_log_stdout": false,
"multiple_tqdm": true,
"print_hypernet_extra": false,
"list_hidden_files": true,
"disable_mmap_load_safetensors": false,
"hide_ldm_prints": true,
"api_enable_requests": true,
"api_forbid_local_requests": true,
"api_useragent": "",
"unload_models_when_training": true,
"pin_memory": false,
"save_optimizer_state": false,
"save_training_settings_to_txt": true,
"dataset_filename_word_regex": "",
"dataset_filename_join_string": " ",
"training_image_repeats_per_epoch": 1,
"training_write_csv_every": 500,
"training_xattention_optimizations": false,
"training_enable_tensorboard": false,
"training_tensorboard_save_images": false,
"training_tensorboard_flush_every": 120,
"sd_model_checkpoint": "SDXL\\copaxTimelessxlSDXL1_v9.safetensors [c967070428]",
"sd_checkpoints_limit": 2,
"sd_checkpoints_keep_in_cpu": true,
"sd_checkpoint_cache": 0,
"sd_unet": "Automatic",
"enable_quantization": false,
"enable_emphasis": true,
"enable_batch_seeds": true,
"comma_padding_backtrack": 20,
"CLIP_stop_at_last_layers": 1,
"upcast_attn": false,
"randn_source": "GPU",
"tiling": false,
"hires_fix_refiner_pass": "second pass",
"sdxl_crop_top": 0,
"sdxl_crop_left": 0,
"sdxl_refiner_low_aesthetic_score": 2.5,
"sdxl_refiner_high_aesthetic_score": 7.0,
"sd_vae_explanation": "VAE is a neural network that transforms a standard RGB\nimage into latent space representation and back. Latent space representation is what stable diffusion is working on during sampling\n(i.e. when the progress bar is between empty and full). For txt2img, VAE is used to create a resulting image after the sampling is finished.\nFor img2img, VAE is used to process user's input image before the sampling, and to create an image after sampling.",
"sd_vae_checkpoint_cache": 1,
"sd_vae": "sdxl__mod_vae.safetensors",
"sd_vae_overrides_per_model_preferences": true,
"auto_vae_precision": true,
"sd_vae_encode_method": "Full",
"sd_vae_decode_method": "Full",
"inpainting_mask_weight": 1.0,
"initial_noise_multiplier": 1.0,
"img2img_extra_noise": 0.0,
"img2img_color_correction": false,
"img2img_fix_steps": false,
"img2img_background_color": "#ffffff",
"img2img_editor_height": 720,
"img2img_sketch_default_brush_color": "#ffffff",
"img2img_inpaint_mask_brush_color": "#ffffff",
"img2img_inpaint_sketch_default_brush_color": "#ffffff",
"return_mask": false,
"return_mask_composite": false,
"cross_attention_optimization": "xformers",
"s_min_uncond": 0.0,
"token_merging_ratio": 0.0,
"token_merging_ratio_img2img": 0.0,
"token_merging_ratio_hr": 0.0,
"pad_cond_uncond": false,
"persistent_cond_cache": true,
"batch_cond_uncond": true,
"use_old_emphasis_implementation": false,
"use_old_karras_scheduler_sigmas": false,
"no_dpmpp_sde_batch_determinism": false,
"use_old_hires_fix_width_height": false,
"dont_fix_second_order_samplers_schedule": false,
"hires_fix_use_firstpass_conds": false,
"use_old_scheduling": false,
"interrogate_keep_models_in_memory": false,
"interrogate_return_ranks": false,
"interrogate_clip_num_beams": 1,
"interrogate_clip_min_length": 24,
"interrogate_clip_max_length": 48,
"interrogate_clip_dict_limit": 1500,
"interrogate_clip_skip_categories": [],
"interrogate_deepbooru_score_threshold": 0.5,
"deepbooru_sort_alpha": true,
"deepbooru_use_spaces": true,
"deepbooru_escape": true,
"deepbooru_filter_tags": "",
"extra_networks_show_hidden_directories": true,
"extra_networks_hidden_models": "When searched",
"extra_networks_default_multiplier": 1.0,
"extra_networks_card_width": 192.0,
"extra_networks_card_height": 192.0,
"extra_networks_card_text_scale": 1.0,
"extra_networks_card_show_desc": true,
"extra_networks_add_text_separator": " ",
"ui_extra_networks_tab_reorder": "",
"textual_inversion_print_at_load": false,
"textual_inversion_add_hashes_to_infotext": true,
"sd_hypernetwork": "None",
"localization": "None",
"gradio_theme": "derekzen/stardust",
"gradio_themes_cache": true,
"gallery_height": "",
"return_grid": true,
"do_not_show_images": false,
"send_seed": true,
"send_size": true,
"js_modal_lightbox": true,
"js_modal_lightbox_initially_zoomed": true,
"js_modal_lightbox_gamepad": false,
"js_modal_lightbox_gamepad_repeat": 250,
"show_progress_in_title": true,
"samplers_in_dropdown": true,
"dimensions_and_batch_together": true,
"keyedit_precision_attention": 0.1,
"keyedit_precision_extra": 0.05,
"keyedit_delimiters": ".,\\/!?%^*;:{}=`~()",
"keyedit_move": true,
"quicksettings_list": [
"sd_model_checkpoint",
"sd_vae",
"randn_source",
"CLIP_stop_at_last_layers"
],
"ui_tab_order": [],
"hidden_tabs": [],
"ui_reorder_list": [],
"hires_fix_show_sampler": false,
"hires_fix_show_prompts": false,
"disable_token_counters": false,
"add_model_hash_to_info": true,
"add_model_name_to_info": true,
"add_user_name_to_info": false,
"add_version_to_infotext": true,
"disable_weights_auto_swap": true,
"infotext_styles": "Apply if any",
"show_progressbar": true,
"live_previews_enable": true,
"live_previews_image_format": "png",
"show_progress_grid": false,
"show_progress_every_n_steps": 10,
"show_progress_type": "Approx NN",
"live_preview_allow_lowvram_full": false,
"live_preview_content": "Combined",
"live_preview_refresh_period": 1000.0,
"live_preview_fast_interrupt": false,
"hide_samplers": [],
"eta_ddim": 0.1,
"eta_ancestral": 0.96,
"ddim_discretize": "uniform",
"s_churn": 0,
"s_tmin": 0,
"s_tmax": 0,
"s_noise": 1,
"k_sched_type": "Automatic",
"sigma_min": 0.0,
"sigma_max": 0.0,
"rho": 0.0,
"eta_noise_seed_delta": 0,
"always_discard_next_to_last_sigma": false,
"sgm_noise_multiplier": false,
"uni_pc_variant": "bh1",
"uni_pc_skip_type": "time_uniform",
"uni_pc_order": 3,
"uni_pc_lower_order_final": true,
"postprocessing_enable_in_main_ui": [],
"postprocessing_operation_order": [],
"upscaling_max_images_in_cache": 5,
"disabled_extensions": [
"sd-dynamic-thresholding"
],
"disable_all_extensions": "none",
"restore_config_state_file": "",
"sd_checkpoint_hash": "1601484d4bbc8869d904ee23ff45e4f210f8e782a5e1b826d89c4af0ef8fe219",
"ldsr_steps": 100,
"ldsr_cached": false,
"SCUNET_tile": 256,
"SCUNET_tile_overlap": 8,
"SWIN_tile": 192,
"SWIN_tile_overlap": 8,
"lora_functional": false,
"sd_lora": "None",
"lora_preferred_name": "Alias from file",
"lora_add_hashes_to_infotext": true,
"lora_show_all": false,
"lora_hide_unknown_for_versions": [],
"lora_in_memory_limit": 0,
"extra_options_txt2img": [],
"extra_options_img2img": [],
"extra_options_cols": 1,
"extra_options_accordion": false,
"canvas_hotkey_zoom": "Alt",
"canvas_hotkey_adjust": "Ctrl",
"canvas_hotkey_move": "F",
"canvas_hotkey_fullscreen": "S",
"canvas_hotkey_reset": "R",
"canvas_hotkey_overlap": "O",
"canvas_show_tooltip": true,
"canvas_auto_expand": true,
"canvas_blur_prompt": false,
"canvas_disabled_functions": [
"Overlap"
],
"styles_ui": "select-list",
"enable_styleselector_by_default": true,
"aesthetic_scorer_enabled": true,
"aesthetic_scorer_clip_model": "ViT-L/14",
"model_preview_xd_name_matching": "Loose",
"model_preview_xd_limit_sizing": true,
"image_browser_active_tabs": "txt2img, img2img, txt2img-grids, img2img-grids, Extras, Favorites, Others, All, Maintenance",
"image_browser_hidden_components": [],
"image_browser_with_subdirs": true,
"image_browser_preload": false,
"image_browser_copy_image": false,
"image_browser_delete_message": true,
"image_browser_txt_files": true,
"image_browser_debug_level": "0 - none",
"image_browser_delete_recycle": true,
"image_browser_scan_exif": true,
"image_browser_mod_shift": false,
"image_browser_mod_ctrl_shift": false,
"image_browser_ranking_pnginfo": false,
"image_browser_page_columns": 6.0,
"image_browser_page_rows": 6.0,
"image_browser_pages_perload": 20.0,
"image_browser_height_auto": false,
"image_browser_use_thumbnail": false,
"image_browser_thumbnail_size": 192.0,
"image_browser_swipe": false,
"image_browser_img_tooltips": true,
"image_browser_show_progress": true,
"image_browser_info_add": true,
"dp_ignore_whitespace": false,
"dp_write_raw_template": true,
"dp_write_prompts_to_file": false,
"dp_parser_variant_start": "{",
"dp_parser_variant_end": "}",
"dp_parser_wildcard_wrap": "__",
"dp_limit_jinja_prompts": false,
"dp_auto_purge_cache": false,
"dp_wildcard_manager_no_dedupe": false,
"dp_wildcard_manager_no_sort": false,
"dp_wildcard_manager_shuffle": false,
"dp_magicprompt_default_model": "Gustavosta/MagicPrompt-Stable-Diffusion",
"dp_magicprompt_batch_size": 1,
"ad_max_models": 2,
"ad_extra_models_dir": "",
"ad_save_previews": false,
"ad_save_images_before": false,
"ad_only_seleted_scripts": true,
"ad_script_names": "dynamic_prompting,dynamic_thresholding,wildcard_recursive,wildcards,lora_block_weight",
"ad_bbox_sortby": "None",
"ad_same_seed_for_each_tap": false,
"use_aria2": true,
"disable_dns": false,
"show_log": false,
"split_aria2": 64,
"aria2_flags": "",
"insert_sub": true,
"use_LORA": false,
"unpack_zip": false,
"hide_early_access": true,
"custom_api_key": "7dc31e1708b211cc2cf65dd698e2b9bb",
"page_header": false,
"queue_paused": false,
"queue_button_hide_checkpoint": true,
"queue_button_placement": "Under Generate button",
"queue_ui_placement": "As a tab",
"queue_history_retention_days": "30 days",
"queue_automatic_requeue_failed_task": false,
"queue_grid_page_size": 0,
"queue_keyboard_shortcut": "Ctrl+KeyE",
"update_log": true,
"control_net_detectedmap_dir": "detected_maps",
"control_net_models_path": "",
"control_net_modules_path": "",
"control_net_unit_count": 2,
"control_net_model_cache_size": 1,
"control_net_inpaint_blur_sigma": 7,
"control_net_no_high_res_fix": false,
"control_net_no_detectmap": false,
"control_net_detectmap_autosaving": false,
"control_net_allow_script_control": false,
"control_net_sync_field_args": true,
"controlnet_show_batch_images_in_ui": false,
"controlnet_increment_seed_during_batch": false,
"controlnet_disable_control_type": false,
"controlnet_disable_openpose_edit": false,
"controlnet_ignore_noninpaint_mask": false,
"image_location": "",
"Checkpoint_subfolder": "\\SDXL",
"VAE_subfolder": "None",
"Wildcards_subfolder": "None",
"regprp_debug": false,
"regprp_hidepmask": false,
"dot_subfolders": true,
"LORA_subfolder": "\\SDXL",
"LoCon_subfolder": "None",
"TextualInversion_subfolder": "None",
"Poses_subfolder": "None",
"Controlnet_subfolder": "None",
"Hypernetwork_subfolder": "None",
"MotionModule_subfolder": "None",
"SWINIR_upscale_subfolder": "None",
"REALESRGAN_upscale_subfolder": "None",
"GFPGAN_upscale_subfolder": "None",
"BSRGAN_upscale_subfolder": "None",
"ESRGAN_upscale_subfolder": "None",
"AestheticGradient_subfolder": "None",
"Workflows_subfolder": "None",
"Other_subfolder": "None",
"model_preview_xd_column_view": false,
"freeu_png_info_auto_enable": true,
"SWIN_torch_compile": false,
"controlnet_disable_photopea_edit": false,
"controlnet_photopea_warning": true,
"controlnet_input_thumbnail": true,
"save_api_info": false,
"auto_save_all_img": false,
"use_local_html": false,
"video_playback": true,
"sub_image_location": true,
"local_path_in_html": false,
"save_to_custom": false,
"model_preview_xd_cache_images_civitai_info": false,
"queue_completion_action": "Do nothing",
"save_images_replace_action": "Replace",
"notification_audio": false,
"notification_volume": 100,
"auto_backcompat": true,
"use_downcasted_alpha_bar": true,
"extra_networks_dir_button_function": false,
"extra_networks_card_order_field": "Path",
"extra_networks_card_order": "Ascending",
"extra_networks_tree_view_default_enabled": false,
"lora_not_found_warning_console": false,
"lora_not_found_gradio_warning": false,
"pad_cond_uncond_v0": false,
"fp8_storage": "Disable",
"cache_fp16_weight": false,
"sd_noise_schedule": "Default",
"emphasis": "Original",
"enable_prompt_comments": true,
"auto_vae_precision_bfloat16": false,
"img2img_batch_show_results_limit": 32,
"overlay_inpaint": true,
"sd_webui_modal_lightbox_icon_opacity": 1,
"sd_webui_modal_lightbox_toolbar_opacity": 0.9,
"add_vae_name_to_info": true,
"add_vae_hash_to_info": true,
"infotext_skip_pasting": [],
"js_live_preview_in_modal_lightbox": false,
"keyedit_delimiters_whitespace": [
"Tab",
"Carriage Return",
"Line Feed"
],
"include_styles_into_token_counters": true,
"compact_prompt_box": false,
"sd_checkpoint_dropdown_use_short": false,
"txt2img_settings_accordion": false,
"img2img_settings_accordion": false,
"interrupt_after_current": true,
"enable_console_prompts": false,
"enable_upscale_progressbar": true,
"dump_stacks_on_signal": false,
"postprocessing_existing_caption_action": "Ignore",
"dat_enabled_models": [
"DAT x2",
"DAT x3",
"DAT x4"
],
"DAT_tile": 192,
"DAT_tile_overlap": 8,
"canvas_hotkey_shrink_brush": "Q",
"canvas_hotkey_grow_brush": "W"
},
"Startup": {
"total": 19.44104266166687,
"records": {
"initial startup": 0.03125166893005371,
"prepare environment/checks": 0.015638113021850586,
"prepare environment/git version info": 0.13801980018615723,
"prepare environment/torch GPU test": 2.4381277561187744,
"prepare environment/clone repositores": 0.26950693130493164,
"prepare environment": 3.262460470199585,
"launcher": 0.01563239097595215,
"import torch": 7.685915946960449,
"import gradio": 0.0,
"setup paths": 0.0,
"import ldm": 0.0,
"import sgm": 0.0,
"initialize shared": 0.16928672790527344,
"other imports": 0.733590841293335,
"opts onchange": 0.01561284065246582,
"setup SD model": 0.0,
"setup codeformer": 0.0,
"setup gfpgan": 0.022176504135131836,
"set samplers": 0.0,
"list extensions": 0.0,
"restore config state file": 0.0,
"list SD models": 4.959932088851929,
"list localizations": 0.0,
"load scripts/custom_code.py": 0.015624284744262695,
"load scripts/img2imgalt.py": 0.0,
"load scripts/loopback.py": 0.0,
"load scripts/outpainting_mk_2.py": 0.0156252384185791,
"load scripts/poor_mans_outpainting.py": 0.0,
"load scripts/postprocessing_caption.py": 0.0,
"load scripts/postprocessing_codeformer.py": 0.0,
"load scripts/postprocessing_create_flipped_copies.py": 0.0,
"load scripts/postprocessing_focal_crop.py": 0.006505012512207031,
"load scripts/postprocessing_gfpgan.py": 0.0,
"load scripts/postprocessing_split_oversized.py": 0.0,
"load scripts/postprocessing_upscale.py": 0.0,
"load scripts/processing_autosized_crop.py": 0.0,
"load scripts/prompt_matrix.py": 0.0,
"load scripts/prompts_from_file.py": 0.0,
"load scripts/sd_upscale.py": 0.0,
"load scripts/xyz_grid.py": 0.0,
"load scripts/ldsr_model.py": 0.6176230907440186,
"load scripts/lora_script.py": 0.21627569198608398,
"load scripts/scunet_model.py": 0.03126072883605957,
"load scripts/swinir_model.py": 0.03772592544555664,
"load scripts/hotkey_config.py": 0.0,
"load scripts/extra_options_section.py": 0.0,
"load scripts/legacy_preprocessors.py": 0.015695571899414062,
"load scripts/preprocessor_inpaint.py": 0.015593767166137695,
"load scripts/preprocessor_marigold.py": 0.16933274269104004,
"load scripts/preprocessor_normalbae.py": 0.015630483627319336,
"load scripts/preprocessor_recolor.py": 0.0,
"load scripts/forge_reference.py": 0.0,
"load scripts/preprocessor_revision.py": 0.0,
"load scripts/preprocessor_tile.py": 0.0,
"load scripts/forge_controllllite.py": 0.0156252384185791,
"load scripts/controlnet.py": 0.3851773738861084,
"load scripts/xyz_grid_support.py": 0.0,
"load scripts/sd_forge_controlnet_example.py": 0.0,
"load scripts/forge_dynamic_thresholding.py": 0.0,
"load scripts/forge_fooocus_inpaint.py": 0.0,
"load scripts/forge_freeu.py": 0.0,
"load scripts/forge_hypertile.py": 0.015641450881958008,
"load scripts/forge_ipadapter.py": 0.0,
"load scripts/kohya_hrfix.py": 0.0,
"load scripts/forge_latent_modifier.py": 0.0,
"load scripts/forge_multidiffusion.py": 0.015647172927856445,
"load scripts/forge_photomaker.py": 0.0,
"load scripts/forge_sag.py": 0.0,
"load scripts/forge_stylealign.py": 0.0,
"load scripts/forge_svd.py": 0.053380489349365234,
"load scripts/forge_z123.py": 0.03128480911254883,
"load scripts/soft_inpainting.py": 0.0,
"load scripts/comments.py": 0.046907901763916016,
"load scripts/refiner.py": 0.0,
"load scripts/seed.py": 0.0,
"load scripts": 1.7205569744110107,
"load upscalers": 0.0,
"refresh VAE": 0.03789544105529785,
"refresh textual inversion templates": 0.0,
"scripts list_optimizers": 0.03120708465576172,
"scripts list_unets": 0.0,
"reload hypernetworks": 0.0,
"initialize extra networks": 0.031194686889648438,
"scripts before_ui_callback": 0.0,
"create ui": 0.4391639232635498,
"gradio launch": 0.4857490062713623,
"add APIs": 0.0,
"app_started_callback/lora_script.py": 0.0,
"app_started_callback/controlnet.py": 0.0,
"app_started_callback": 0.0
}
},
"Packages": [
"absl-py==2.1.0",
"accelerate==0.21.0",
"addict==2.4.0",
"aenum==3.1.15",
"aiofiles==23.2.1",
"aiohttp==3.9.3",
"aiosignal==1.3.1",
"albumentations==1.3.1",
"altair==5.2.0",
"antlr4-python3-runtime==4.9.3",
"anyio==3.7.1",
"astunparse==1.6.3",
"async-timeout==4.0.3",
"attrs==23.2.0",
"basicsr==1.4.2",
"beautifulsoup4==4.12.3",
"blendmodes==2022",
"certifi==2024.2.2",
"cffi==1.16.0",
"chardet==5.2.0",
"charset-normalizer==3.3.2",
"clean-fid==0.1.35",
"click==8.1.7",
"clip==1.0",
"colorama==0.4.6",
"coloredlogs==15.0.1",
"colorlog==6.8.2",
"contourpy==1.2.0",
"cssselect2==0.7.0",
"cycler==0.12.1",
"cython==3.0.8",
"deprecation==2.1.0",
"depth-anything==2024.1.22.0",
"diffusers==0.25.0",
"dm-tree==0.1.8",
"dynamicprompts==0.30.4",
"easydict==1.12",
"einops==0.4.1",
"embreex==2.17.7.post4",
"exceptiongroup==1.2.0",
"facexlib==0.3.0",
"fake-useragent==1.4.0",
"fastapi==0.94.0",
"ffmpy==0.3.2",
"filelock==3.13.1",
"filterpy==1.4.5",
"flatbuffers==23.5.26",
"fonttools==4.48.1",
"frozenlist==1.4.1",
"fsspec==2024.2.0",
"ftfy==6.1.3",
"future==0.18.3",
"fvcore==0.1.5.post20221221",
"gast==0.5.4",
"gitdb==4.0.11",
"gitpython==3.1.32",
"google-pasta==0.2.0",
"gradio-client==0.5.0",
"gradio==3.41.2",
"greenlet==3.0.3",
"grpcio==1.60.1",
"h11==0.12.0",
"h5py==3.10.0",
"handrefinerportable==2024.2.12.0",
"httpcore==0.15.0",
"httpx==0.24.1",
"huggingface-hub==0.20.3",
"humanfriendly==10.0",
"idna==3.6",
"imageio==2.34.0",
"importlib-metadata==7.0.1",
"importlib-resources==6.1.1",
"inflection==0.5.1",
"insightface==0.7.3",
"iopath==0.1.9",
"jinja2==3.1.3",
"joblib==1.3.2",
"jsonmerge==1.8.0",
"jsonschema-specifications==2023.12.1",
"jsonschema==4.21.1",
"keras-nightly==3.0.4.dev2024021403",
"kiwisolver==1.4.5",
"kornia==0.6.7",
"lark==1.1.2",
"lazy-loader==0.3",
"libclang==16.0.6",
"lightning-utilities==0.10.1",
"llvmlite==0.42.0",
"lmdb==1.4.1",
"lxml==5.1.0",
"mapbox-earcut==1.0.1",
"markdown-it-py==3.0.0",
"markdown==3.5.2",
"markupsafe==2.1.5",
"matplotlib==3.8.3",
"mdurl==0.1.2",
"mediapipe==0.10.9",
"ml-dtypes==0.3.2",
"mpmath==1.3.0",
"multidict==6.0.5",
"namex==0.0.7",
"networkx==3.2.1",
"numba==0.59.0",
"numpy==1.26.2",
"omegaconf==2.2.3",
"onnx==1.15.0",
"onnxruntime-gpu==1.17.0",
"onnxruntime==1.17.0",
"open-clip-torch==2.20.0",
"opencv-contrib-python==4.9.0.80",
"opencv-python-headless==4.9.0.80",
"opencv-python==4.9.0.80",
"opt-einsum==3.3.0",
"orjson==3.9.14",
"packaging==23.2",
"pandas==2.2.0",
"piexif==1.1.3",
"pillow==9.5.0",
"pip==22.2.1",
"platformdirs==4.2.0",
"portalocker==2.8.2",
"prettytable==3.9.0",
"protobuf==3.20.3",
"psutil==5.9.5",
"py-cpuinfo==9.0.0",
"pycollada==0.8",
"pycparser==2.21",
"pydantic==1.10.14",
"pydub==0.25.1",
"pygments==2.17.2",
"pyparsing==3.1.1",
"pyreadline3==3.4.1",
"python-dateutil==2.8.2",
"python-multipart==0.0.9",
"pytorch-lightning==1.9.4",
"pytz==2024.1",
"pywavelets==1.5.0",
"pywin32==306",
"pyyaml==6.0.1",
"qudida==0.0.4",
"referencing==0.33.0",
"regex==2023.12.25",
"reportlab==4.1.0",
"requests==2.31.0",
"resize-right==0.0.2",
"rich==13.7.0",
"rpds-py==0.18.0",
"rtree==1.2.0",
"safetensors==0.4.2",
"scikit-image==0.21.0",
"scikit-learn==1.4.0",
"scipy==1.12.0",
"seaborn==0.13.2",
"semantic-version==2.10.0",
"send2trash==1.8.2",
"sentencepiece==0.1.99",
"setuptools==63.2.0",
"shapely==2.0.2",
"six==1.16.0",
"smmap==5.0.1",
"sniffio==1.3.0",
"sounddevice==0.4.6",
"soupsieve==2.5",
"spandrel==0.1.6",
"sqlalchemy==2.0.27",
"starlette==0.26.1",
"svg.path==6.3",
"svglib==1.5.1",
"sympy==1.12",
"tabulate==0.9.0",
"tb-nightly==2.16.0a20240212",
"tensorboard-data-server==0.7.2",
"tensorflow-io-gcs-filesystem==0.31.0",
"termcolor==2.4.0",
"tf-keras-nightly==2.17.0.dev2024021510",
"tf-nightly-intel==2.17.0.dev20240214",
"tf-nightly==2.17.0.dev20240214",
"thop==0.1.1.post2209072238",
"threadpoolctl==3.3.0",
"tifffile==2024.2.12",
"timm==0.9.12",
"tinycss2==1.2.1",
"tokenizers==0.13.3",
"tomesd==0.1.3",
"tomli==2.0.1",
"toolz==0.12.1",
"torch==2.1.2+cu121",
"torchdiffeq==0.2.3",
"torchmetrics==1.3.1",
"torchsde==0.2.6",
"torchvision==0.16.2+cu121",
"tqdm==4.66.2",
"trampoline==0.1.2",
"transformers==4.30.2",
"trimesh==4.1.3",
"typing-extensions==4.9.0",
"tzdata==2024.1",
"ultralytics==8.1.14",
"urllib3==2.2.0",
"uvicorn==0.27.1",
"vhacdx==0.0.5",
"wcwidth==0.2.13",
"webencodings==0.5.1",
"websockets==11.0.3",
"werkzeug==3.0.1",
"wheel==0.42.0",
"wrapt==1.16.0",
"xformers==0.0.23.post1",
"xxhash==3.4.1",
"yacs==0.1.8",
"yapf==0.40.2",
"yarl==1.9.4",
"zipp==3.17.0",
"zipunicode==1.1.1"
]
}

### Console logs

```Shell
https://pastebin.com/2hMRBsxL
```

### Additional information

The NVIDIA driver is set to prevent the GPU from using system RAM if VRAM runs out. Also see below for comparable runs from AUTO1111 (first and second runs with refiner, third without)

```
venv "C:\Applications\StableDiffusion\Automatic1111\webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug 1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.7.0
Commit hash: cf2772fab0af5573da775e7437e6acdca424f26e
CUDA 11.8
Launching Web UI with arguments: --xformers --autolaunch --port 24278 --gradio-auth xx:xx
Style database not found: C:\Applications\StableDiffusion\Automatic1111\webui\styles.csv
16:33:52 - ReActor - STATUS - Running v0.7.0-a1 on Device: CUDA
[-] ADetailer initialized. version: 24.1.2, num models: 19
CivitAI Browser+: Aria2 RPC started
Using sqlite file: C:\Applications\StableDiffusion\Automatic1111\webui\extensions\sd-webui-agent-scheduler\task_scheduler.sqlite3
ControlNet preprocessor location: M:\Backups\Stable Diffusion Items\controlnet downloads
2024-02-16 16:33:54,142 - ControlNet - INFO - ControlNet v1.1.440
2024-02-16 16:33:54,211 - ControlNet - INFO - ControlNet v1.1.440
[sd-webui-freeu] Controlnet support: *enabled*
Loading weights [c967070428] from C:\Applications\StableDiffusion\Automatic1111\webui\models\Stable-diffusion\SDXL\copaxTimelessxlSDXL1_v9.safetensors
2024-02-16 16:33:55,106 - ControlNet - INFO - ControlNet UI callback registered.
Creating model from config: C:\Applications\StableDiffusion\Automatic1111\webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Loading VAE weights specified in settings: C:\Applications\StableDiffusion\Automatic1111\webui\models\VAE\SDXL\sdxl__mod_vae.safetensors
Applying attention optimization: xformers... done.
Model loaded in 4.8s (load weights from disk: 0.9s, create model: 0.5s, apply weights to model: 2.8s, load VAE: 0.1s, calculate empty prompt: 0.2s).
Running on local URL: http://127.0.0.1:24278

To create a public link, set `share=True` in `launch()`.
Startup time: 22.1s (prepare environment: 6.8s, import torch: 2.7s, import gradio: 0.7s, setup paths: 0.6s, initialize shared: 0.2s, other imports: 0.5s, setup codeformer: 0.2s, list SD models: 0.2s, load scripts: 4.2s, create ui: 5.2s, gradio launch: 0.5s, app_started_callback: 0.2s).
80%|█████████████████████████████████████████████████████████████████▌ | 32/40 [00:25<00:06, 1.29it/s]Loading model SDXL\newrealityxlAllInOne_21.safetensors [1601484d4b] (2 out of 2) | 32/40 [00:24<00:06, 1.29it/s]
Loading weights [1601484d4b] from C:\Applications\StableDiffusion\Automatic1111\webui\models\Stable-diffusion\SDXL\newrealityxlAllInOne_21.safetensors
Creating model from config: C:\Applications\StableDiffusion\Automatic1111\webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Loading VAE weights specified in settings: cached sdxl__mod_vae.safetensors
Applying attention optimization: xformers... done.
Model loaded in 72.2s (load weights from disk: 8.0s, create model: 0.5s, apply weights to model: 63.1s, load VAE: 0.2s, move model to device: 0.1s, calculate empty prompt: 0.2s).
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [01:46<00:00, 2.66s/it]
Loading CLiP model ViT-L/14 ███████████████████████████████████████████████████████████| 40/40 [01:45<00:00, 2.63s/it]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 40/40 [01:51<00:00, 2.78s/it]
Using already loaded model SDXL\copaxTimelessxlSDXL1_v9.safetensors [c967070428]: done in 3.7s (send model to cpu: 1.8s, send model to device: 1.9s)
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:31<00:00, 1.28it/s]
Using already loaded model SDXL\copaxTimelessxlSDXL1_v9.safetensors [c967070428]: done in 5.8s (send model to cpu: 3.1s, send model to device: 2.7s)
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:31<00:00, 1.28it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 40/40 [00:32<00:00, 1.25it/s]
100%|██████████████████████████████████████████████████████████████████████████████████| 40/40 [00:31<00:00, 1.28it/s]
Total progress: 100%|██████████████████████████████████████████████████████████████████| 40/40 [00:32<00:00, 1.25it/s]

```

Start by launching Forge with the reported checkpoint, refiner, and checkpoint-cache settings, then reproduce a prompt run with and without the refiner while comparing load times. Trace the checkpoint loading entry point and verify whether configured device and CPU retention is respected; done means refiner runs no longer repeatedly reload checkpoints and timing is stable.

Written by the indexing model from the issue text.