DeepSeek-R1-Distill-Qwen-14B/32B sym_int4 could not run lm-scaler-vllm:0.26.0-b1/2

IMAGE:
Llm-scaler-vllm:0.26.0-b1/b2
Model:
DeepSeek-R1-Distill-Qwen-14B/32B sym_int4
loading models and quantization error:

vllm cmd:
root@edgeai10:/llm-scaler/vllm# TORCH_LLM_ALLREDUCE=1 VLLM_USE_V1=1 W_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 -m vllm.entrypoints.openai.api_server --model /llm/models/DeepSeek-R1-Distill-Qwen-14B --dtype=float16 --enforce-eager --port 8000 --block-size 64 --gpu-memory-util 0.9 --trust-remote-code --max_num_batched_tokens=4096 --max_model_len 20000 --quantization sym_int4 -tp=4

or
root@edgeai10:/llm-scaler/vllm# TORCH_LLM_ALLREDUCE=1 VLLM_USE_V1=1 W_LONG_MAX_MODEL_LEN=1 VLLM_WORKER_MULTIPROC_METHOD=spawn python3 -m vllm.entrypoints.openai.api_server --model /llm/models/DeepSeek-R1-Distill-Qwen-32B --dtype=float16 --enforce-eager --port 8000 --block-size 64 --gpu-memory-util 0.9 --trust-remote-code --max_num_batched_tokens=4096 --max_model_len 20000 --quantization sym_int4 -tp=4

**error:**
RuntimeError: sym_int4 requires a complete checkpoint load before runtime quantization ，
**more info:**

14B error:

ing weights by their parent layer when reloading.
(Worker_TP0 pid=2175) WARNING 09-10 14:14:59 [layerwise.py:210] Allocating 0.3 MB of device memory to buffers to load ['QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear'] layers. This extra memory usage can be avoided by ordering weights by their parent layer when reloading.
(Worker_TP0 pid=2175) WARNING 09-10 14:14:59 [layerwise.py:210] Allocating 0.3 MB of device memory to buffers to load ['QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear'] layers. This extra memory usage can be avoided by ordering weights by their parent layer when reloading.
(Worker_TP0 pid=2175) WARNING 09-10 14:14:59 [layerwise.py:210] Allocating 0.3 MB of device memory to buffers to load ['QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear'] layers. This extra memory usage can be avoided by ordering weights by their parent layer when reloading.
(Worker_TP0 pid=2175) WARNING 09-10 14:14:59 [layerwise.py:210] Allocating 0.3 MB of device memory to buffers to load ['QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear'] layers. This extra memory usage can be avoided by ordering weights by their parent layer when reloading.
(Worker_TP0 pid=2175) WARNING 09-10 14:14:59 [layerwise.py:210] Allocating 0.3 MB of device memory to buffers to load ['QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear', 'QKVParallelLinear'] layers. This extra memory usage can be avoided by ordering weights by their parent layer when reloading.
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00, 1.00s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:04<00:00, 1.13s/it]
(Worker_TP0 pid=2175)
(Worker_TP0 pid=2175) INFO 09-10 14:14:59 [default_loader.py:430] Loading weights took 4.52 seconds
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] WorkerProc failed to start.
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] Traceback (most recent call last):
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 868, in worker_main
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] worker = WorkerProc(*args, **kwargs)
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] return func(*args, **kwargs)
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 637, in __init__
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] self.worker.load_model()
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 431, in load_model
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu/model_runner.py", line 311, in load_model
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] self.model = model_loader.load_model(
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] return func(*args, **kwargs)
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 78, in load_model
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] finalize_layerwise_processing(model, model_config)
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/reload/layerwise.py", line 277, in finalize_layerwise_processing
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] _layerwise_process(layer, info)
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/reload/layerwise.py", line 366, in _layerwise_process
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] quant_method.process_weights_after_loading(layer)
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/sym_int4.py", line 320, in process_weights_after_loading
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] _require_complete_layerwise_load(layer, ("weight",))
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/sym_int4.py", line 438, in _require_complete_layerwise_load
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] raise RuntimeError(
(Worker_TP1 pid=2176) ERROR 09-10 14:14:59 [multiproc_executor.py:901] RuntimeError: sym_int4 requires a complete checkpoint load before runtime quantization (loaded 9176576 of 9176832 elements)
(EngineCore pid=2096) INFO 09-10 14:14:59 [multiproc_executor.py:429] [shutdown] Executor: waiting for worker exit count=4
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] WorkerProc failed to start.
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] Traceback (most recent call last):
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 868, in worker_main
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] worker = WorkerProc(*args, **kwargs)
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] return func(*args, **kwargs)
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 637, in __init__
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] self.worker.load_model()
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 431, in load_model
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu/model_runner.py", line 311, in load_model
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] self.model = model_loader.load_model(
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] return func(*args, **kwargs)
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 78, in load_model
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] finalize_layerwise_processing(model, model_config)
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/reload/layerwise.py", line 277, in finalize_layerwise_processing
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] _layerwise_process(layer, info)
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/reload/layerwise.py", line 366, in _layerwise_process
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] quant_method.process_weights_after_loading(layer)
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/sym_int4.py", line 320, in process_weights_after_loading
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] _require_complete_layerwise_load(layer, ("weight",))
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/sym_int4.py", line 438, in _require_complete_layerwise_load
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] raise RuntimeError(
(Worker_TP3 pid=2178) ERROR 09-10 14:14:59 [multiproc_executor.py:901] RuntimeError: sym_int4 requires a complete checkpoint load before runtime quantization (loaded 9176576 of 9176832 elements)
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] WorkerProc failed to start.
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] Traceback (most recent call last):
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 868, in worker_main
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] worker = WorkerProc(*args, **kwargs)
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] return func(*args, **kwargs)
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 637, in __init__
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] self.worker.load_model()
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 431, in load_model
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu/model_runner.py", line 311, in load_model
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] self.model = model_loader.load_model(
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] return func(*args, **kwargs)
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 78, in load_model
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] finalize_layerwise_processing(model, model_config)
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/reload/layerwise.py", line 277, in finalize_layerwise_processing
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] _layerwise_process(layer, info)
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/reload/layerwise.py", line 366, in _layerwise_process
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] quant_method.process_weights_after_loading(layer)
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/sym_int4.py", line 320, in process_weights_after_loading
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] _require_complete_layerwise_load(layer, ("weight",))
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/sym_int4.py", line 438, in _require_complete_layerwise_load
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] raise RuntimeError(
(Worker_TP0 pid=2175) ERROR 09-10 14:14:59 [multiproc_executor.py:901] RuntimeError: sym_int4 requires a complete checkpoint load before runtime quantization (loaded 9176576 of 9176832 elements)
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] WorkerProc failed to start.
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] Traceback (most recent call last):
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 868, in worker_main
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] worker = WorkerProc(*args, **kwargs)
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] return func(*args, **kwargs)
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 637, in __init__
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] self.worker.load_model()
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 431, in load_model
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu/model_runner.py", line 311, in load_model
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] self.model = model_loader.load_model(
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] return func(*args, **kwargs)
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/base_loader.py", line 78, in load_model
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] finalize_layerwise_processing(model, model_config)
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/reload/layerwise.py", line 277, in finalize_layerwise_processing
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] _layerwise_process(layer, info)
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/model_loader/reload/layerwise.py", line 366, in _layerwise_process
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] quant_method.process_weights_after_loading(layer)
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/sym_int4.py", line 320, in process_weights_after_loading
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] _require_complete_layerwise_load(layer, ("weight",))
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/sym_int4.py", line 438, in _require_complete_layerwise_load
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] raise RuntimeError(
(Worker_TP2 pid=2177) ERROR 09-10 14:14:59 [multiproc_executor.py:901] RuntimeError: sym_int4 requires a complete checkpoint load before runtime quantization (loaded 9176576 of 9176832 elements)
(EngineCore pid=2096) INFO 09-10 14:15:01 [multiproc_executor.py:436] [shutdown] Executor: all workers exited gracefully
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] EngineCore failed to start.
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] Traceback (most recent call last):
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1299, in run_engine_core
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] return func(*args, **kwargs)
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1065, in __init__
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] super().__init__(
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 125, in __init__
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] self.model_executor = executor_class(vllm_config)
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 108, in __init__
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] super().__init__(vllm_config)
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] return func(*args, **kwargs)
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 109, in __init__
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] self._init_executor()
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 201, in _init_executor
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 765, in wait_for_ready
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] raise e from None
(EngineCore pid=2096) ERROR 09-10 14:15:01 [core.py:1330] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(EngineCore pid=2096) Process EngineCore:
(EngineCore pid=2096) Traceback (most recent call last):
(EngineCore pid=2096) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=2096) self.run()

Reproduce the failure with the two vllm.entrypoints.openai.api_server commands for the 14B and 32B models. Start with vllm/model_executor/layers/quantization/sym_int4.py and vllm/model_executor/model_loader/reload/layerwise.py, following the complete-load check and the reported element mismatch. Done means both models start with --quantization sym_int4 without the RuntimeError.

Written by the indexing model from the issue text.