Unable to run Qwen3.6-35B-A3B FP8 on 2x B70s using `llm-scaler-vllm:0.14.0-b8.2`

Error:
```
RuntimeError: level_zero backend failed with error: 40 (UR_RESULT_ERROR_OUT_OF_RESOURCES)
```

System:
```
root@ayrtons-ubuntu-server:/llm# uname -r
6.17.0-22-generic

root@ayrtons-ubuntu-server:/llm# sycl-ls
[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero V2, Intel(R) Graphics [0xe223] 20.2.0 [1.14.37435+12]
[level_zero:gpu][level_zero:1] Intel(R) oneAPI Unified Runtime over Level-Zero V2, Intel(R) Graphics [0xe223] 20.2.0 [1.14.37435+12]
[opencl:cpu][opencl:0] Intel(R) OpenCL, AMD EPYC-Milan-v3 Processor OpenCL 3.0 (Build 0) [2026.20.1.0.12_160000]
[opencl:gpu][opencl:1] Intel(R) OpenCL Graphics, Intel(R) Graphics [0xe223] OpenCL 3.0 NEO [26.09.37435.12]
[opencl:gpu][opencl:2] Intel(R) OpenCL Graphics, Intel(R) Graphics [0xe223] OpenCL 3.0 NEO [26.09.37435.12]

root@ayrtons-ubuntu-server:/llm# free -h
total used free shared buff/cache available
Mem: 91Gi 2.8Gi 55Gi 4.7Mi 34Gi 89Gi
Swap: 63Gi 42Mi 63Gi

root@ayrtons-ubuntu-server:/llm# printenv | grep VLLM
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
VLLM_WORKER_MULTIPROC_METHOD=spawn
VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1
VLLM_QUANTIZE_Q40_LIB=/usr/local/lib/python3.12/dist-packages/vllm_int4_for_multi_arc.so
VLLM_TARGET_DEVICE=xpu
```

Logs

```bash
root@ayrtons-ubuntu-server:/llm# vllm serve ~/.cache/huggingface/Qwen3.6-35B-A3B --served-model-name Qwen --dtype=float16 --enforce-eager --port 8000 --host 0.0.0.0 --trust-remote-code --disable-sliding-window --gpu-memory-util=0.9 --max-num-batched-tokens=8192 --disable-log-requests --max-model-len=8192 --block-size 64 --quantization fp8 -tp=2 --kv-cache-dtype fp8
[W425 19:41:19.403743730 OperatorEntry.cpp:208] Warning: Warning only once for all operators, other operators may also be overridden.
Overriding a previously registered kernel for the same operator and the same dispatch key
operator: aten::geometric_(Tensor(a!) self, float p, *, Generator? generator=None) -> Tensor(a!)
registered at /pytorch/build/aten/src/ATen/RegisterSchema.cpp:6
dispatch key: XPU
previous kernel: registered at /pytorch/aten/src/ATen/VmapModeRegistrations.cpp:36
new kernel: registered at /root/workspace/frameworks.ai.pytorch.ipex-gpu/build/Release/csrc/gpu/csrc/gpu/xpu/ATen/RegisterXPU_0.cpp:172 (function operator())
(APIServer pid=1640) INFO 04-25 19:41:31 [api_server.py:1272] vLLM API server version 0.14.1.dev0+gb17039bcc.d20260422
(APIServer pid=1640) INFO 04-25 19:41:31 [utils.py:263] non-default args: {'model_tag': '/root/.cache/huggingface/Qwen3.6-35B-A3B', 'host': '0.0.0.0', 'model': '/root/.cache/huggingface/Qwen3.6-35B-A3B', 'trust_remote_code': True, 'dtype': 'float16', 'max_model_len': 8192, 'quantization': 'fp8', 'enforce_eager': True, 'disable_sliding_window': True, 'served_model_name': ['Qwen'], 'tensor_parallel_size': 2, 'block_size': 64, 'kv_cache_dtype': 'fp8', 'max_num_batched_tokens': 8192}
(APIServer pid=1640) [transformers] The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=1640) [transformers] The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=1640) INFO 04-25 19:41:31 [model.py:533] Resolved architecture: Qwen3_5MoeForConditionalGeneration
(APIServer pid=1640) WARNING 04-25 19:41:31 [_logger.py:68] Casting torch.bfloat16 to torch.float16.
(APIServer pid=1640) INFO 04-25 19:41:31 [model.py:1549] Using max model len 8192
(APIServer pid=1640) INFO 04-25 19:41:31 [cache.py:206] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor.
(APIServer pid=1640) INFO 04-25 19:41:31 [scheduler.py:229] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=1640) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(APIServer pid=1640) INFO 04-25 19:41:32 [config.py:479] Setting attention block size to 1088 tokens to ensure that attention page size is >= mamba page size.
(APIServer pid=1640) INFO 04-25 19:41:32 [config.py:503] Padding mamba page size by 1.49% to ensure that mamba page size and attention page size are exactly equal.
(APIServer pid=1640) INFO 04-25 19:41:32 [vllm.py:636] Asynchronous scheduling is disabled.
(APIServer pid=1640) WARNING 04-25 19:41:32 [_logger.py:68] Enforce eager set, overriding optimization level to -O0
[W425 19:41:38.936512114 OperatorEntry.cpp:208] Warning: Warning only once for all operators, other operators may also be overridden.
Overriding a previously registered kernel for the same operator and the same dispatch key
operator: aten::geometric_(Tensor(a!) self, float p, *, Generator? generator=None) -> Tensor(a!)
registered at /pytorch/build/aten/src/ATen/RegisterSchema.cpp:6
dispatch key: XPU
previous kernel: registered at /pytorch/aten/src/ATen/VmapModeRegistrations.cpp:36
new kernel: registered at /root/workspace/frameworks.ai.pytorch.ipex-gpu/build/Release/csrc/gpu/csrc/gpu/xpu/ATen/RegisterXPU_0.cpp:172 (function operator())
(EngineCore_DP0 pid=1663) INFO 04-25 19:41:47 [core.py:97] Initializing a V1 LLM engine (v0.14.1.dev0+gb17039bcc.d20260422) with config: model='/root/.cache/huggingface/Qwen3.6-35B-A3B', speculative_config=None, tokenizer='/root/.cache/huggingface/Qwen3.6-35B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=fp8, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=xpu, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=Qwen, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': None, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None}
(EngineCore_DP0 pid=1663) WARNING 04-25 19:41:47 [_logger.py:68] Reducing Torch parallelism from 2 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
[W425 19:41:53.445119716 OperatorEntry.cpp:208] Warning: Warning only once for all operators, other operators may also be overridden.
Overriding a previously registered kernel for the same operator and the same dispatch key
operator: aten::geometric_(Tensor(a!) self, float p, *, Generator? generator=None) -> Tensor(a!)
registered at /pytorch/build/aten/src/ATen/RegisterSchema.cpp:6
dispatch key: XPU
previous kernel: registered at /pytorch/aten/src/ATen/VmapModeRegistrations.cpp:36
new kernel: registered at /root/workspace/frameworks.ai.pytorch.ipex-gpu/build/Release/csrc/gpu/csrc/gpu/xpu/ATen/RegisterXPU_0.cpp:172 (function operator())
[W425 19:41:53.446129754 OperatorEntry.cpp:208] Warning: Warning only once for all operators, other operators may also be overridden.
Overriding a previously registered kernel for the same operator and the same dispatch key
operator: aten::geometric_(Tensor(a!) self, float p, *, Generator? generator=None) -> Tensor(a!)
registered at /pytorch/build/aten/src/ATen/RegisterSchema.cpp:6
dispatch key: XPU
previous kernel: registered at /pytorch/aten/src/ATen/VmapModeRegistrations.cpp:36
new kernel: registered at /root/workspace/frameworks.ai.pytorch.ipex-gpu/build/Release/csrc/gpu/csrc/gpu/xpu/ATen/RegisterXPU_0.cpp:172 (function operator())
[transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
[transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
INFO 04-25 19:42:05 [parallel_state.py:1214] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:51859 backend=xccl
INFO 04-25 19:42:05 [parallel_state.py:1214] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:51859 backend=xccl
INFO 04-25 19:42:05 [parallel_state.py:1425] rank 1 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 1, EP rank 1
INFO 04-25 19:42:05 [parallel_state.py:1425] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0
2026:04:25-19:42:05:( 1682) |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi)
2026:04:25-19:42:05:( 1681) |CCL_WARN| value of CCL_ATL_TRANSPORT changed to be ofi (default:mpi)
2026:04:25-19:42:05:( 1681) |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL
2026:04:25-19:42:05:( 1682) |CCL_WARN| could not get local_idx/count from environment variables, trying to get them from ATL
2026:04:25-19:42:06:( 1681) |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices
2026:04:25-19:42:06:( 1682) |CCL_WARN| topology recognition shows PCIe connection between devices. If this is not correct, you can disable topology recognition, with CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0. This will assume XeLinks across devices
[transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
[transformers] The `use_fast` parameter is deprecated and will be removed in a future version. Use `backend="torchvision"` instead of `use_fast=True`, or `backend="pil"` instead of `use_fast=False`.
(Worker_TP0 pid=1681) INFO 04-25 19:42:13 [gpu_model_runner.py:3811] Starting to load model /root/.cache/huggingface/Qwen3.6-35B-A3B...
(Worker_TP0 pid=1681) INFO 04-25 19:42:14 [xpu.py:106] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(Worker_TP0 pid=1681) INFO 04-25 19:42:14 [xpu.py:103] Using backend AttentionBackendEnum.IPEX for vit attention
(Worker_TP0 pid=1681) INFO 04-25 19:42:14 [mm_encoder_attention.py:89] Using AttentionBackendEnum.IPEX for MMEncoderAttention.
(Worker_TP0 pid=1681) INFO 04-25 19:42:14 [fp8.py:126] DeepGEMM is disabled because the platform does not support it.
(Worker_TP0 pid=1681) INFO 04-25 19:42:14 [fp8.py:149] Using Triton backend for FP8 MoE
(Worker_TP0 pid=1681) INFO 04-25 19:42:14 [xpu.py:53] Setting VLLM_KV_CACHE_LAYOUT to 'NHD' for XPU; only NHD layout is supported by XPU attention kernels.
(Worker_TP0 pid=1681) INFO 04-25 19:42:14 [xpu.py:79] Using Flash Attention backend.
Loading safetensors checkpoint shards: 0% Completed | 0/26 [00:00
(APIServer pid=1640) sys.exit(main())
(APIServer pid=1640) ^^^^^^
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=1640) args.dispatch_function(args)
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 60, in cmd
(APIServer pid=1640) uvloop.run(run_server(args))
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=1640) return __asyncio.run(
(APIServer pid=1640) ^^^^^^^^^^^^^^
(APIServer pid=1640) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=1640) return runner.run(main)
(APIServer pid=1640) ^^^^^^^^^^^^^^^^
(APIServer pid=1640) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1640) return self._loop.run_until_complete(task)
(APIServer pid=1640) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1640) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1640) return await main
(APIServer pid=1640) ^^^^^^^^^^
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1319, in run_server
(APIServer pid=1640) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1338, in run_server_worker
(APIServer pid=1640) async with build_async_engine_client(
(APIServer pid=1640) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1640) return await anext(self.gen)
(APIServer pid=1640) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 173, in build_async_engine_client
(APIServer pid=1640) async with build_async_engine_client_from_engine_args(
(APIServer pid=1640) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1640) return await anext(self.gen)
(APIServer pid=1640) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 214, in build_async_engine_client_from_engine_args
(APIServer pid=1640) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1640) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 205, in from_vllm_config
(APIServer pid=1640) return cls(
(APIServer pid=1640) ^^^^
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 132, in __init__
(APIServer pid=1640) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1640) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 122, in make_async_mp_client
(APIServer pid=1640) return AsyncMPClient(*client_args)
(APIServer pid=1640) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 824, in __init__
(APIServer pid=1640) super().__init__(
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 479, in __init__
(APIServer pid=1640) with launch_core_engines(vllm_config, executor_class, log_stats) as (
(APIServer pid=1640) File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=1640) next(self.gen)
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 921, in launch_core_engines
(APIServer pid=1640) wait_for_engine_startup(
(APIServer pid=1640) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 980, in wait_for_engine_startup
(APIServer pid=1640) raise RuntimeError(
(APIServer pid=1640) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore_DP0': 1}
````