Synchronous model loads run inside the shared workflows thread pool, stalling all concurrent workflow runs (latency collapse with idle GPU/CPU)

Summary

On a GPU inference server (roboflow-inference-server-gpu, build of 2026-06-05 / v1.3.0 era) serving multi-model Workflows under sustained production traffic, we observed a latency collapse mode where p50 latency of all workflow endpoints rises 3–10x while accelerator duty cycle stays ≤13% and CPU stays under 25% of limit. Raw /infer/* endpoints on the same pods remain fast. Root cause analysis points at a structural issue: synchronous model loads (3–19s each) execute inside the shared workflows ThreadPoolExecutor, starving step execution for every concurrent workflow run in the process.

Environment

• gpu_http:app via uvicorn, 1 worker, defaults: HTTP_API_SHARED_WORKFLOWS_THREAD_POOL_ENABLED=True (16 workers), WORKFLOWS_MAX_CONCURRENT_STEPS=4, MAX_ACTIVE_MODELS=20, MEMORY_FREE_THRESHOLD=0.20
• Workload: ~6 workflow variants chaining 3–5 models each; working set ~12 model/version combos (~600MB VRAM each) on NVIDIA L4
• Horizontal autoscaling on request rate; replica churn redistributes traffic across pods regularly

Observed

• Model loaded: ... load_time= events: p50 ≈ 5s, p90 ≈ 11s, occurring continuously in steady state (~200–700/hr fleet-wide) because the model working set + MEMORY_FREE_THRESHOLD VRAM guard keeps the WithFixedSizeCache manager evicting (reason=memory_pressure) and re-loading
• When autoscaler churn raises per-pod cache-miss rates, workflow p50/p90 explodes (0.8s → 4–8s p90 across all workflow endpoints uniformly) while GPU duty cycle ≤13% and CPU well under limit — classic heavy-tailed-service queueing on a software bottleneck (E[S²] dominated by inline loads), not a hardware one
• /infer/* on the same pods degrades only mildly (~2x p90 at worst) — consistent with it not using the shared workflows pool, only the model-manager lock

Code path

• inference/core/interfaces/http/http_api.py (~L1193): one process-global self.shared_thread_pool_executor = ThreadPoolExecutor(max_workers=16) is created and passed to every ExecutionEngine.init(...) (~L1293)
• Model blocks execute as steps in that pool; a cold model triggers add_model → download + deserialize + VRAM upload (3–19s) inside a step worker, while also holding the WithFixedSizeCache queue lock for eviction bookkeeping (inference/core/managers/decorators/fixed_size_cache.py), so concurrent runs both lose a pool worker and contend on the lock
• Under churn, several in-flight loads consume a large fraction of the 16 workers; queued steps back up; every workflow run slows; in-flight requests then pile up to whatever the ASGI concurrency limit is, producing 503 load shedding while the GPU is idle

Asks

1. Isolate model loading from step execution — run loads on a dedicated loader executor (or async path) so a cold model only blocks requests that need that model, never the shared step pool
2. First-class model pre-warming: a way to declare the deployment's model set to be loaded at startup (and pinned), with readiness gated on warm-up, so steady-state inline loads approach zero (the _pinned_models machinery in WithFixedSizeCache looks close but isn't exposed/documented for this)
3. Hysteresis / actively-used protection on memory_pressure evictions — under the VRAM threshold, LRU eviction of models that are in active rotation creates a permanent evict↔reload cycle
4. Observability: counters/histograms for model load duration, cache miss rate, and shared-pool queue depth would have made this diagnosable from metrics instead of log archaeology

Happy to provide more detail on the analysis.

Start by tracing the shared_thread_pool_executor and ExecutionEngine.init calls in inference/core/interfaces/http/http_api.py, then inspect cache locking and eviction in inference/core/managers/decorators/fixed_size_cache.py. Reproduce the pool starvation with synchronous model loads and review existing workflow tests. Done means loads no longer occupy the shared step pool, with coverage for concurrent workflow execution and the requested loading behavior.