how to speed up model switching?

Environment
OS: Linux
Python: 3.12 (conda env turbodiffusion)
PyTorch: 2.7.1+cu126
CUDA: Driver Version: 550.163.01 CUDA Version: 12.4
GPU: 1x GPU with 39.38GB VRAM (visible as CUDA device 2)
Run command:
CUDA_VISIBLE_DEVICES=2 python api_server.py

What I’m trying to do
My GPU VRAM is not enough to keep both I2V models (high-noise + low-noise) resident at the same time, so I’m trying to:
Load both models once at startup
Keep them on CPU RAM (model.cpu())
During sampling, move the active model to GPU (model.cuda())
Switch to the other model at the boundary by offloading one back to CPU and loading the other to GPU

Problem
The CPU→GPU switch is very slow.
From logs, moving high noise model to GPU takes ~53 seconds, and switching to low noise model takes ~52 seconds too:
[15:05:32] Loading high noise model to GPU...
[15:06:25] High noise model loaded: GPU Memory: 24.44GB allocated, 24.48GB reserved
[15:07:01] Switching to low noise model...
[15:08:06] Low noise model loaded: GPU Memory: 24.44GB allocated, 24.46GB reserved
So each .cuda() takes ~50s even though model VRAM delta is about:
Before: ~10.85GB
After: ~24.44GB
Delta ≈ 13.6GB, which is much slower than expected PCIe throughput.
Expected behavior
I expected .cpu() -> .cuda() to be much faster (a few seconds range), or at least there should be an officially recommended way to speed up model switching/offload.
Additional observation
Startup loads weights to CPU fine
/proc/$pid/status shows VmRSS ~34GB, VmSwap 0
ulimit -l is large enough (32989920 KB)
Models are on /dev/shm, so disk I/O shouldn’t be the bottleneck
The sampling itself is fast; the slow part is only model .cuda() and model switching.
Questions

Is this behavior expected for this repo / model architecture?

Is there a recommended method to reduce CPU→GPU transfer time?

Use pinned memory (pin_memory) for parameters/buffers?

Use non_blocking=True with to(device)?

Avoid .cpu() / .cuda() full-model move; instead load partially or keep some layers on GPU?

Use safetensors / different checkpoint format / load strategy?

Is there an official multi-GPU strategy for I2V model switching?

e.g. keep high model on GPU0 and low model on GPU1 and move only activations?

Any profiling tips to confirm whether the bottleneck is PCIe bandwidth, page faults, pinned memory, or PyTorch allocator behavior?|

Start at api_server.py and reproduce the high-noise and low-noise model switches described in the logs, profiling the .cpu() and .cuda() operations and their memory behavior. Done would be a confirmed explanation of the transfer cost and a repository-supported model-switching approach, with measurements showing its effect.