[Bug] Qwen3.5-397B-A17B-FP8 fails to load: expert_data shape mismatch (2048 vs 4096) at _load_w13, even with TP=1 + AVX2 backend

### Reminder

- [x] I have read the above rules and searched the existing issues.

### System Info

## Environment
- kt-kernel: 0.6.2.post1
- sglang-kt: 0.6.2.post1
- transformers: 4.57.1
- CPU: Intel Xeon Platinum 8352V (Ice Lake-SP, 2x36 cores, 2 NUMA)
- CPU instruction sets: AVX-512F/BW/DQ/CD/VL/IFMA/VBMI/VBMI2/VNNI/VPOPCNTDQ/BITALG (NO AVX512_BF16, NO AMX)
- GPU: 8x RTX 4090 24GB
- RAM: 503 GiB
- Model: Qwen/Qwen3.5-397B-A17B-FP8
- OS: Ubuntu 22.04.5 LTS (GNU/Linux 5.15.0-176-generic x86_64)

### Reproduction

## Reproduction
Same error with all of:
- TP=1 / TP=8
- KT_KERNEL_CPU_VARIANT=avx2 / avx512_vbmi (auto)
- kt-num-gpu-experts=0 / 1 / 2 / 96 / 128

Minimal command (TP=1, AVX2 backend, follows AVX2-Tutorial.md FP8 example):

# export KT_KERNEL_CPU_VARIANT=avx2
export KT_KERNEL_DEBUG=1
python -c "import kt_kernel; print(kt_kernel.__cpu_variant__)"

[kt-kernel] Using environment override: avx2
[kt-kernel] Selected CPU variant: avx2
[kt-kernel] Multi-variant avx2 not found, using single-variant build
[kt-kernel] Loading avx2 from: /root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/kt_kernel/kt_kernel_ext.cpython-312-x86_64-linux-gnu.so
[kt-kernel] Successfully loaded AVX2 variant
[kt-kernel] Extension module loaded: kt_kernel_ext
avx2
#
# pkill -f sglang || true
sleep 3

unset PYTORCH_CUDA_ALLOC_CONF
export PYTORCH_ALLOC_CONF=expandable_segments:True
export CUDA_VISIBLE_DEVICES=0
export KT_KERNEL_CPU_VARIANT=avx2
export SGLANG_ENABLE_JIT_DEEPGEMM=0

python -m sglang.launch_server \
--host 127.0.0.1 --port 30000 \
--model /br_file/Qwen3.5-397B-A17B-FP8 \
--kt-weight-path /br_file/Qwen3.5-397B-A17B-FP8 \
--kt-cpuinfer 72 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 2 \
--kt-method FP8 \
--kt-gpu-prefill-token-threshold 400 \
--attention-backend triton \
--trust-remote-code \
--mem-fraction-static 0.85 \
--chunked-prefill-size 4096 \
--max-running-requests 1 \
--max-total-tokens 32000 \
--enable-mixed-chunk \
--tensor-parallel-size 1 \
--disable-shared-experts-fusion \
--served-model-name qwen35-397b

## Error
[FP8SafeTensorLoader] Detected format: deepseek
[FP8SafeTensorLoader] Detected scale format: block-wise (weight_scale_inv)
[FP8SafeTensorLoader] Detected VL model ← suspicious: this is text-only inference

File "sglang/srt/layers/moe/fused_moe_triton/layer.py", line 497, in _load_w13
expert_data.copy_(loaded_weight)
RuntimeError: The size of tensor a (2048) must match the size of tensor b (4096) at non-singleton dimension 1

dim=1 = hidden dimension. Qwen3.5 hidden_size = 4096.
expert_data buffer was allocated as 2048 (= 4096/2), regardless of TP size.

## Hypothesis
FP8SafeTensorLoader detects Qwen3.5-397B-A17B-FP8 as a VL model and may be using
a wrong hidden dimension when sizing GPU expert buffers. Same model + AVX2-Tutorial
FP8 example template fails identically on TP=1 single-GPU.

### Others

# pkill -f sglang || true
sleep 3

unset PYTORCH_CUDA_ALLOC_CONF
export PYTORCH_ALLOC_CONF=expandable_segments:True
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export SGLANG_APPLY_CONFIG_BACKUP=none
export SGLANG_ENABLE_JIT_DEEPGEMM=0

python -m sglang.launch_server \
--host 127.0.0.1 \
--port 30000 \
--model /br_file/Qwen3.5-397B-A17B-FP8 \
--served-model-name qwen35-397b-fp8 \
--kt-weight-path /br_file/Qwen3.5-397B-A17B-FP8 \
--kt-method FP8 \
--kt-cpuinfer 64 \
--kt-threadpool-count 2 \
--kt-num-gpu-experts 1 \
--kt-max-deferred-experts-per-token 2 \
--kt-expert-placement-strategy uniform \
--trust-remote-code \
--attention-backend triton \
--linear-attn-backend triton \
--mem-fraction-static 0.25 \
--chunked-prefill-size 64 \
--max-total-tokens 1024 \
--max-running-requests 1 \
--tensor-parallel-size 8 \
--disable-custom-all-reduce \
--disable-shared-experts-fusion \
--disable-cuda-graph \
--skip-server-warmup \
--watchdog-timeout 3000 \
--log-requests \
--log-requests-level 2 \
--show-time-cost \
--decode-log-interval 1
[2026-05-10 03:14:53] WARNING server_args.py:1849: Disabling overlap schedule since mamba no_buffer is not compatible with overlap schedule, try to use --disable-radix-cache if overlap schedule is necessary
[2026-05-10 03:14:54] server_args=ServerArgs(model_path='/br_file/Qwen3.5-397B-A17B-FP8', tokenizer_path='/br_file/Qwen3.5-397B-A17B-FP8', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='127.0.0.1', port=30000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=True, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.25, max_running_requests=1, max_queued_requests=None, max_total_tokens=1024, chunked_prefill_size=64, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, device='cuda', tp_size=8, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=638896193, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=3000.0, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, model_checksum=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=True, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=True, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, extra_metric_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=1, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, admin_api_key=None, served_model_name='qwen35-397b-fp8', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', attn_cp_size=1, moe_dp_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='flashinfer_cutlass', nsa_prefill_backend=None, nsa_decode_backend=None, disable_flashinfer_autotune=False, mamba_backend='triton', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, enable_aiter_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype=None, mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, linear_attn_backend='triton', linear_attn_decode_backend=None, linear_attn_prefill_backend=None, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', disable_hicache_numa_detect=False, hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_hisparse=False, hierarchical_sparse_attention_extra_config=None, enable_lmcache=False, kt_weight_path='/br_file/Qwen3.5-397B-A17B-FP8', kt_method='FP8', kt_cpuinfer=64, kt_threadpool_count=2, kt_numa_nodes=None, kt_num_gpu_experts=1, kt_gpu_experts_ratio=None, kt_max_deferred_experts_per_token=2, kt_gpu_prefill_token_threshold=None, record_kt_gpu_expert_distribution=False, kt_enable_dynamic_expert_update=False, kt_expert_placement_strategy='uniform', dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=80, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80], disable_cuda_graph=True, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=True, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=True, enable_mixed_chunk=False, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=64, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=True, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, enable_return_indexer_topk=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, nsa_prefill_cp_mode='round-robin-split', enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, limit_mm_data_per_request=None, enable_mm_global_cache=False, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None)
[2026-05-10 03:14:54] Ignore import error when loading sglang.srt.multimodal.processors.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/transformers/__init__.py)
[2026-05-10 03:14:56] Using default HuggingFace chat template with detected content format: openai
[2026-05-10 03:15:04 TP7] Mamba selective_state_update backend initialized: triton
[2026-05-10 03:15:04 TP7] Init torch distributed begin.
[2026-05-10 03:15:04 TP6] Mamba selective_state_update backend initialized: triton
[2026-05-10 03:15:04 TP5] Mamba selective_state_update backend initialized: triton
[2026-05-10 03:15:04 TP6] Init torch distributed begin.
[2026-05-10 03:15:04 TP5] Init torch distributed begin.
[2026-05-10 03:15:04 TP2] Mamba selective_state_update backend initialized: triton
[2026-05-10 03:15:04 TP2] Init torch distributed begin.
[2026-05-10 03:15:04 TP4] Mamba selective_state_update backend initialized: triton
[2026-05-10 03:15:04 TP0] Mamba selective_state_update backend initialized: triton
[2026-05-10 03:15:04 TP3] Mamba selective_state_update backend initialized: triton
[2026-05-10 03:15:04 TP0] Init torch distributed begin.
[2026-05-10 03:15:04 TP3] Init torch distributed begin.
[2026-05-10 03:15:04 TP4] Init torch distributed begin.
[2026-05-10 03:15:04 TP1] Mamba selective_state_update backend initialized: triton
[2026-05-10 03:15:04 TP1] Init torch distributed begin.
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 0 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 1 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 3 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 2 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[2026-05-10 03:15:07 TP0] sglang is using nccl==2.27.5
[Gloo] Rank 4 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 5 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 7 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 6 is connected to 7 peer ranks. Expected number of connected peer ranks is : 7
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2026-05-10 03:15:07 TP6] Init torch distributed ends. elapsed=3.10 s, mem usage=0.18 GB
[2026-05-10 03:15:07 TP5] Init torch distributed ends. elapsed=3.10 s, mem usage=0.18 GB
[2026-05-10 03:15:07 TP0] Init torch distributed ends. elapsed=2.83 s, mem usage=0.18 GB
[2026-05-10 03:15:07 TP7] Init torch distributed ends. elapsed=3.17 s, mem usage=0.18 GB
[2026-05-10 03:15:07 TP4] Init torch distributed ends. elapsed=2.82 s, mem usage=0.18 GB
[2026-05-10 03:15:07 TP3] Init torch distributed ends. elapsed=2.83 s, mem usage=0.18 GB
[2026-05-10 03:15:07 TP1] Init torch distributed ends. elapsed=2.78 s, mem usage=0.18 GB
[2026-05-10 03:15:07 TP2] Init torch distributed ends. elapsed=3.02 s, mem usage=0.18 GB
[2026-05-10 03:15:07 TP2] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4: No module named 'tilelang'
[2026-05-10 03:15:07 TP2] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4_nextn: No module named 'tilelang'
[2026-05-10 03:15:07 TP5] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4: No module named 'tilelang'
[2026-05-10 03:15:07 TP6] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4: No module named 'tilelang'
[2026-05-10 03:15:07 TP5] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4_nextn: No module named 'tilelang'
[2026-05-10 03:15:07 TP1] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4: No module named 'tilelang'
[2026-05-10 03:15:07 TP6] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4_nextn: No module named 'tilelang'
[2026-05-10 03:15:07 TP3] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4: No module named 'tilelang'
[2026-05-10 03:15:07 TP1] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4_nextn: No module named 'tilelang'
[2026-05-10 03:15:07 TP3] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4_nextn: No module named 'tilelang'
[2026-05-10 03:15:07 TP7] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4: No module named 'tilelang'
[2026-05-10 03:15:07 TP7] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4_nextn: No module named 'tilelang'
[2026-05-10 03:15:07 TP0] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4: No module named 'tilelang'
[2026-05-10 03:15:07 TP0] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4_nextn: No module named 'tilelang'
[2026-05-10 03:15:07 TP4] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4: No module named 'tilelang'
[2026-05-10 03:15:07 TP4] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4_nextn: No module named 'tilelang'
[2026-05-10 03:15:08 TP2] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP2] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr_nextn: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP2] In import_model_classes: Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/transformers/__init__.py)
[2026-05-10 03:15:08 TP5] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP5] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr_nextn: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP5] In import_model_classes: Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/transformers/__init__.py)
[2026-05-10 03:15:08 TP3] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP1] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP3] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr_nextn: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP6] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP3] In import_model_classes: Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/transformers/__init__.py)
[2026-05-10 03:15:08 TP1] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr_nextn: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP6] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr_nextn: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP1] In import_model_classes: Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/transformers/__init__.py)
[2026-05-10 03:15:08 TP7] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP6] In import_model_classes: Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/transformers/__init__.py)
[2026-05-10 03:15:08 TP7] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr_nextn: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP7] In import_model_classes: Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/transformers/__init__.py)
[2026-05-10 03:15:08 TP0] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP0] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr_nextn: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP0] In import_model_classes: Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/transformers/__init__.py)
[2026-05-10 03:15:08 TP4] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP4] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr_nextn: No module named 'transformers.models.glm_ocr'
[2026-05-10 03:15:08 TP4] In import_model_classes: Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/transformers/__init__.py)
[2026-05-10 03:15:09 TP2] Load weight begin. avail mem=22.85 GB
[2026-05-10 03:15:09 TP5] Load weight begin. avail mem=22.85 GB
[2026-05-10 03:15:09 TP7] Load weight begin. avail mem=22.85 GB
[2026-05-10 03:15:09 TP2] Multimodal attention backend not set. Use triton_attn.
[2026-05-10 03:15:09 TP2] Using triton_attn as multimodal attention backend.
[2026-05-10 03:15:09 TP6] Load weight begin. avail mem=22.85 GB
[2026-05-10 03:15:09 TP1] Load weight begin. avail mem=22.85 GB
`torch_dtype` is deprecated! Use `dtype` instead!
[2026-05-10 03:15:09 TP2] Using kt_num_gpu_experts=1 per layer, total GPU experts: 60 (= 1 × 60 MoE layers)
[2026-05-10 03:15:09 TP5] Multimodal attention backend not set. Use triton_attn.
[2026-05-10 03:15:09 TP5] Using triton_attn as multimodal attention backend.
[2026-05-10 03:15:09 TP7] Multimodal attention backend not set. Use triton_attn.
[2026-05-10 03:15:09 TP7] Using triton_attn as multimodal attention backend.
`torch_dtype` is deprecated! Use `dtype` instead!
[2026-05-10 03:15:09 TP5] Using kt_num_gpu_experts=1 per layer, total GPU experts: 60 (= 1 × 60 MoE layers)
[2026-05-10 03:15:09 TP6] Multimodal attention backend not set. Use triton_attn.
[2026-05-10 03:15:09 TP6] Using triton_attn as multimodal attention backend.
[2026-05-10 03:15:09 TP0] Load weight begin. avail mem=22.85 GB
[2026-05-10 03:15:09 TP1] Multimodal attention backend not set. Use triton_attn.
[2026-05-10 03:15:09 TP1] Using triton_attn as multimodal attention backend.
[2026-05-10 03:15:09 TP0] Detected fp8 checkpoint.
[2026-05-10 03:15:09 TP4] Load weight begin. avail mem=22.85 GB
`torch_dtype` is deprecated! Use `dtype` instead!
[2026-05-10 03:15:09 TP7] Using kt_num_gpu_experts=1 per layer, total GPU experts: 60 (= 1 × 60 MoE layers)
`torch_dtype` is deprecated! Use `dtype` instead!
[2026-05-10 03:15:09 TP6] Using kt_num_gpu_experts=1 per layer, total GPU experts: 60 (= 1 × 60 MoE layers)
`torch_dtype` is deprecated! Use `dtype` instead!
[2026-05-10 03:15:09 TP1] Using kt_num_gpu_experts=1 per layer, total GPU experts: 60 (= 1 × 60 MoE layers)
[2026-05-10 03:15:09 TP3] Load weight begin. avail mem=22.85 GB
[2026-05-10 03:15:09 TP0] Multimodal attention backend not set. Use triton_attn.
[2026-05-10 03:15:09 TP0] Using triton_attn as multimodal attention backend.
[2026-05-10 03:15:09 TP4] Multimodal attention backend not set. Use triton_attn.
[2026-05-10 03:15:09 TP4] Using triton_attn as multimodal attention backend.
`torch_dtype` is deprecated! Use `dtype` instead!
[2026-05-10 03:15:09 TP4] Using kt_num_gpu_experts=1 per layer, total GPU experts: 60 (= 1 × 60 MoE layers)
`torch_dtype` is deprecated! Use `dtype` instead!
[2026-05-10 03:15:09 TP0] Using kt_num_gpu_experts=1 per layer, total GPU experts: 60 (= 1 × 60 MoE layers)
[2026-05-10 03:15:09 TP0] Using uniform strategy for GPU expert placement
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 0 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 1 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 2 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 3 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 4 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 5 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 6 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 7 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 8 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 9 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 10 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 11 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 12 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 13 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 14 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 15 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 16 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 17 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 18 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 19 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 20 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 21 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 22 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 23 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 24 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 25 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 26 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 27 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 28 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 29 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 30 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 31 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 32 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 33 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 34 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 35 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 36 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 37 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 38 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 39 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 40 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 41 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 42 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 43 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 44 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 45 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 46 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 47 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 48 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 49 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 50 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 51 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 52 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 53 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 54 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 55 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 56 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 57 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 58 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP0] KT GPU experts: layer 59 (MoE) has 1 GPU experts
[2026-05-10 03:15:09 TP3] Multimodal attention backend not set. Use triton_attn.
[2026-05-10 03:15:09 TP3] Using triton_attn as multimodal attention backend.
[2026-05-10 03:15:09 TP0] Generated KT GPU experts masks using 'uniform' strategy: 60 MoE layers (out of 60 total layers) x 512 experts, total GPU experts in MoE layers = 60
[2026-05-10 03:15:09 TP0] [KT] Created shared staging buffer: 0.5 MiB (shape=torch.Size([64, 4096]), dtype=torch.bfloat16)
CPUInfer[0x5588ef0a9f20]: Hello
WorkerPool[0x5588ef0a9df0] 2 subpools, [numa:threads][0:32] [1:32]
===========In NumaPool============
[2026-05-10 03:15:09 TP4] using attn output gate!
[2026-05-10 03:15:09 TP5] using attn output gate!
[2026-05-10 03:15:09 TP6] using attn output gate!
[2026-05-10 03:15:09 TP2] using attn output gate!
`torch_dtype` is deprecated! Use `dtype` instead!
[2026-05-10 03:15:09 TP3] Using kt_num_gpu_experts=1 per layer, total GPU experts: 60 (= 1 × 60 MoE layers)
[2026-05-10 03:15:09 TP3] using attn output gate!
[2026-05-10 03:15:09 TP1] using attn output gate!
[2026-05-10 03:15:09 TP7] using attn output gate!
In Numa Worker Pool at NUMA 0, 32 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 1, 32 threads
[2026-05-10 03:15:10 TP2] Scheduler hit an exception: Traceback (most recent call last):
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 3196, in run_scheduler_process
scheduler = Scheduler(
^^^^^^^^^^
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 367, in __init__
self.init_model_worker()
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 563, in init_model_worker
self.init_tp_model_worker()
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 521, in init_tp_model_worker
self.tp_worker = TpModelWorker(
^^^^^^^^^^^^^^
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 247, in __init__
self._init_model_runner()
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 330, in _init_model_runner
self._model_runner = ModelRunner(
^^^^^^^^^^^^
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 423, in __init__
self.initialize()
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 506, in initialize
self.load_model()
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 1016, in load_model
self.model = self.loader.load_model(
^^^^^^^^^^^^^^^^^^^^^^^
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 677, in load_model
self.load_weights_and_postprocess(
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/model_loader/loader.py", line 686, in load_weights_and_postprocess
model.load_weights(weights)
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/models/qwen3_5.py", line 1314, in load_weights
weight_loader(
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 671, in weight_loader
self._weight_loader_physical(
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 713, in _weight_loader_physical
self._weight_loader_impl(
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 911, in _weight_loader_impl
self._load_model_weight_or_group_weight_scale(
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 405, in _load_model_weight_or_group_weight_scale
self._load_w13(
File "/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 497, in _load_w13
expert_data.copy_(loaded_weight)
RuntimeError: The size of tensor a (2048) must match the size of tensor b (4096) at non-singleton dimension 1

[2026-05-10 03:15:10] Received sigquit from a child process. It usually means the child failed.
Killed

# python -c "import kt_kernel; print(kt_kernel.__version__, kt_kernel.__cpu_variant__)"
pip show kt-kernel
pip show sglang
0.6.2.post1 avx512_vbmi
Name: kt-kernel
Version: 0.6.2.post1
Summary: KT-Kernel: High-performance kernel operations for KTransformers (AMX/AVX/KML optimizations)
Home-page: https://github.com/kvcache-ai
Author: kvcache-ai
Author-email:
License-Expression: Apache-2.0
Location: /root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages
Requires: gguf, httpx, numpy, packaging, pyyaml, rich, safetensors, torch, triton, typer
Required-by:
WARNING: Package(s) not found: sglang

# pip show sglang-kt
python -c "import sglang; print(sglang.__file__, sglang.__version__)"
Name: sglang-kt
Version: 0.6.2.post1
Summary: SGLang is a fast serving framework for large language models and vision language models.
Home-page: https://github.com/sgl-project/sglang
Author:
Author-email:
License:
Location: /root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages
Requires: aiohttp, anthropic, apache-tvm-ffi, blobfile, build, compressed-tensors, cuda-python, datasets, decord2, einops, fastapi, flashinfer_cubin, flashinfer_python, gguf, grpcio, grpcio-health-checking, grpcio-reflection, hf_transfer, huggingface_hub, interegular, IPython, llguidance, modelscope, msgspec, ninja, numpy, nvidia-cutlass-dsl, nvidia-ml-py, openai, openai-harmony, orjson, outlines, packaging, partial_json_parser, pillow, prometheus-client, psutil, py-spy, pybase64, pydantic, python-multipart, pyzmq, quack-kernels, requests, scipy, sentencepiece, setproctitle, sgl-kernel, smg-grpc-proto, soundfile, tiktoken, timm, torch, torch_memory_saver, torchao, torchaudio, torchcodec, torchvision, tqdm, transformers-kt, uvicorn, uvloop, xgrammar
Required-by:
/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/sglang/__init__.py 0.0.0.dev0

# lscpu | grep -oE "avx512[_a-z0-9]*" | sort -u
avx512_bitalg
avx512bw
avx512cd
avx512dq
avx512f
avx512ifma
avx512vbmi
avx512_vbmi2
avx512vl
avx512_vnni
avx512_vpopcntdq

[2026-05-10 06:47:17] WARNING server_args.py:1849: Disabling overlap schedule since mamba no_buffer is not compatible with overlap schedule, try to use --disable-radix-cache if overlap schedule is necessary
[2026-05-10 06:47:18] server_args=ServerArgs(model_path='/br_file/Qwen3.5-397B-A17B-FP8', tokenizer_path='/br_file/Qwen3.5-397B-A17B-FP8', tokenizer_mode='auto', tokenizer_worker_num=1, skip_tokenizer_init=False, load_format='auto', model_loader_extra_config='{}', trust_remote_code=True, context_length=None, is_embedding=False, enable_multimodal=None, revision=None, model_impl='auto', host='127.0.0.1', port=30000, fastapi_root_path='', grpc_mode=False, skip_server_warmup=False, warmups=None, nccl_port=None, checkpoint_engine_wait_weights_before_ready=False, dtype='auto', quantization=None, quantization_param_path=None, kv_cache_dtype='auto', enable_fp32_lm_head=False, modelopt_quant=None, modelopt_checkpoint_restore_path=None, modelopt_checkpoint_save_path=None, modelopt_export_path=None, quantize_and_serve=False, rl_quant_profile=None, mem_fraction_static=0.85, max_running_requests=1, max_queued_requests=None, max_total_tokens=32000, chunked_prefill_size=4096, enable_dynamic_chunking=False, max_prefill_tokens=16384, prefill_max_requests=None, schedule_policy='fcfs', enable_priority_scheduling=False, abort_on_priority_when_disabled=False, schedule_low_priority_values_first=False, priority_scheduling_preemption_threshold=10, schedule_conservativeness=1.0, page_size=1, swa_full_tokens_ratio=0.8, disable_hybrid_swa_memory=False, radix_eviction_policy='lru', enable_prefill_delayer=False, prefill_delayer_max_delay_passes=30, prefill_delayer_token_usage_low_watermark=None, prefill_delayer_forward_passes_buckets=None, prefill_delayer_wait_seconds_buckets=None, device='cuda', tp_size=1, pp_size=1, pp_max_micro_batch_size=None, pp_async_batch_depth=0, stream_interval=1, stream_output=False, random_seed=869522533, constrained_json_whitespace_pattern=None, constrained_json_disable_any_whitespace=False, watchdog_timeout=300, soft_watchdog_timeout=None, dist_timeout=None, download_dir=None, model_checksum=None, base_gpu_id=0, gpu_id_step=1, sleep_on_idle=False, custom_sigquit_handler=None, log_level='info', log_level_http=None, log_requests=False, log_requests_level=2, log_requests_format='text', log_requests_target=None, uvicorn_access_log_exclude_prefixes=[], crash_dump_folder=None, show_time_cost=False, enable_metrics=False, enable_metrics_for_all_schedulers=False, tokenizer_metrics_custom_labels_header='x-custom-labels', tokenizer_metrics_allowed_custom_labels=None, extra_metric_labels=None, bucket_time_to_first_token=None, bucket_inter_token_latency=None, bucket_e2e_request_latency=None, collect_tokens_histogram=False, prompt_tokens_buckets=None, generation_tokens_buckets=None, gc_warning_threshold_secs=0.0, decode_log_interval=40, enable_request_time_stats_logging=False, kv_events_config=None, enable_trace=False, otlp_traces_endpoint='localhost:4317', export_metrics_to_file=False, export_metrics_to_file_dir=None, api_key=None, admin_api_key=None, served_model_name='qwen35-397b', weight_version='default', chat_template=None, hf_chat_template_name=None, completion_template=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, tool_call_parser=None, tool_server=None, sampling_defaults='model', dp_size=1, load_balance_method='round_robin', attn_cp_size=1, moe_dp_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', preferred_sampling_params=None, enable_lora=None, enable_lora_overlap_loading=None, max_lora_rank=None, lora_target_modules=None, lora_paths=None, max_loaded_loras=None, max_loras_per_batch=8, lora_eviction_policy='lru', lora_backend='csgmv', max_lora_chunk_size=16, attention_backend='triton', decode_attention_backend=None, prefill_attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', mm_attention_backend=None, fp8_gemm_runner_backend='auto', fp4_gemm_runner_backend='flashinfer_cutlass', nsa_prefill_backend=None, nsa_decode_backend=None, disable_flashinfer_autotune=False, mamba_backend='triton', speculative_algorithm=None, speculative_draft_model_path=None, speculative_draft_model_revision=None, speculative_draft_load_format=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, speculative_attention_mode='prefill', speculative_draft_attention_backend=None, speculative_moe_runner_backend='auto', speculative_moe_a2a_backend=None, speculative_draft_model_quantization=None, speculative_ngram_min_match_window_size=1, speculative_ngram_max_match_window_size=12, speculative_ngram_min_bfs_breadth=1, speculative_ngram_max_bfs_breadth=10, speculative_ngram_match_type='BFS', speculative_ngram_branch_length=18, speculative_ngram_capacity=10000000, enable_multi_layer_eagle=False, ep_size=1, moe_a2a_backend='none', moe_runner_backend='auto', flashinfer_mxfp4_moe_precision='default', enable_flashinfer_allreduce_fusion=False, enable_aiter_allreduce_fusion=False, deepep_mode='auto', ep_num_redundant_experts=0, ep_dispatch_algorithm=None, init_expert_location='trivial', enable_eplb=False, eplb_algorithm='auto', eplb_rebalance_num_iterations=1000, eplb_rebalance_layers_per_chunk=None, eplb_min_rebalancing_utilization_threshold=1.0, expert_distribution_recorder_mode=None, expert_distribution_recorder_buffer_size=1000, enable_expert_distribution_metrics=False, deepep_config=None, moe_dense_tp_size=None, elastic_ep_backend=None, mooncake_ib_device=None, max_mamba_cache_size=None, mamba_ssm_dtype=None, mamba_full_memory_ratio=0.9, mamba_scheduler_strategy='no_buffer', mamba_track_interval=256, linear_attn_backend='triton', linear_attn_decode_backend=None, linear_attn_prefill_backend=None, enable_hierarchical_cache=False, hicache_ratio=2.0, hicache_size=0, hicache_write_policy='write_through', hicache_io_backend='kernel', hicache_mem_layout='layer_first', disable_hicache_numa_detect=False, hicache_storage_backend=None, hicache_storage_prefetch_policy='best_effort', hicache_storage_backend_extra_config=None, enable_hisparse=False, hierarchical_sparse_attention_extra_config=None, enable_lmcache=False, kt_weight_path='/br_file/Qwen3.5-397B-A17B-FP8', kt_method='FP8', kt_cpuinfer=72, kt_threadpool_count=2, kt_numa_nodes=None, kt_num_gpu_experts=2, kt_gpu_experts_ratio=None, kt_max_deferred_experts_per_token=None, kt_gpu_prefill_token_threshold=400, record_kt_gpu_expert_distribution=False, kt_enable_dynamic_expert_update=False, kt_expert_placement_strategy='uniform', dllm_algorithm=None, dllm_algorithm_config=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, cpu_offload_gb=0, offload_group_size=-1, offload_num_in_group=1, offload_prefetch_step=1, offload_mode='cpu', multi_item_scoring_delimiter=None, disable_radix_cache=False, cuda_graph_max_bs=24, cuda_graph_bs=[1, 2, 4, 8, 12, 16, 24], disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_profile_cuda_graph=False, enable_cudagraph_gc=False, enable_layerwise_nvtx_marker=False, enable_nccl_nvls=False, enable_symm_mem=False, disable_flashinfer_cutlass_moe_fp4_allgather=False, enable_tokenizer_batch_encode=False, disable_tokenizer_batch_decode=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, enable_mscclpp=False, enable_torch_symm_mem=False, disable_overlap_schedule=True, enable_mixed_chunk=True, enable_dp_attention=False, enable_dp_lm_head=False, enable_two_batch_overlap=False, enable_single_batch_overlap=False, tbo_token_distribution_threshold=0.48, enable_torch_compile=False, enable_piecewise_cuda_graph=False, enable_torch_compile_debug_mode=False, torch_compile_max_bs=32, piecewise_cuda_graph_max_tokens=4096, piecewise_cuda_graph_tokens=[4, 8, 12, 16, 20, 24, 28, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 256, 288, 320, 352, 384, 416, 448, 480, 512, 576, 640, 704, 768, 832, 896, 960, 1024, 1280, 1536, 1792, 2048, 2304, 2560, 2816, 3072, 3328, 3584, 3840, 4096], piecewise_cuda_graph_compiler='eager', torchao_config='', enable_nan_detection=False, enable_p2p_check=False, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, triton_attention_split_tile_size=None, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, enable_weights_cpu_backup=False, enable_draft_weights_cpu_backup=False, allow_auto_truncate=False, enable_custom_logit_processor=False, flashinfer_mla_disable_ragged=False, disable_shared_experts_fusion=True, disable_chunked_prefix_cache=False, disable_fast_image_processor=False, keep_mm_feature_on_device=False, enable_return_hidden_states=False, enable_return_routed_experts=False, enable_return_indexer_topk=False, scheduler_recv_interval=1, numa_node=None, enable_deterministic_inference=False, rl_on_policy_target=None, enable_attn_tp_input_scattered=False, enable_nsa_prefill_context_parallel=False, nsa_prefill_cp_mode='round-robin-split', enable_fused_qk_norm_rope=False, enable_precise_embedding_interpolation=False, enable_dynamic_batch_tokenizer=False, dynamic_batch_tokenizer_batch_size=32, dynamic_batch_tokenizer_batch_timeout=0.002, debug_tensor_dump_output_folder=None, debug_tensor_dump_layers=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_transfer_backend='mooncake', disaggregation_bootstrap_port=8998, disaggregation_decode_tp=None, disaggregation_decode_dp=None, disaggregation_prefill_pp=1, disaggregation_ib_device=None, disaggregation_decode_enable_offload_kvcache=False, num_reserved_decode_tokens=512, disaggregation_decode_polling_interval=1, encoder_only=False, language_only=False, encoder_transfer_backend='zmq_to_scheduler', encoder_urls=[], custom_weight_loader=[], weight_loader_disable_mmap=False, remote_instance_weight_loader_seed_instance_ip=None, remote_instance_weight_loader_seed_instance_service_port=None, remote_instance_weight_loader_send_weights_group_ports=None, remote_instance_weight_loader_backend='nccl', remote_instance_weight_loader_start_seed_via_transfer_engine=False, enable_pdmux=False, pdmux_config_path=None, sm_group_num=8, mm_max_concurrent_calls=32, mm_per_request_timeout=10.0, enable_broadcast_mm_inputs_process=False, enable_prefix_mm_cache=False, mm_enable_dp_encoder=False, mm_process_config={}, limit_mm_data_per_request=None, enable_mm_global_cache=False, decrypted_config_file=None, decrypted_draft_config_file=None, forward_hooks=None)
[kt-kernel] Using environment override: avx2
[kt-kernel] Selected CPU variant: avx2
[kt-kernel] Multi-variant avx2 not found, using single-variant build
[kt-kernel] Loading avx2 from: /root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/kt_kernel/kt_kernel_ext.cpython-312-x86_64-linux-gnu.so
[kt-kernel] Successfully loaded AVX2 variant
[kt-kernel] Extension module loaded: kt_kernel_ext
[2026-05-10 06:47:18] Ignore import error when loading sglang.srt.multimodal.processors.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/transformers/__init__.py)
[2026-05-10 06:47:20] Using default HuggingFace chat template with detected content format: openai
[2026-05-10 06:47:28] Mamba selective_state_update backend initialized: triton
[kt-kernel] Using environment override: avx2
[kt-kernel] Selected CPU variant: avx2
[kt-kernel] Multi-variant avx2 not found, using single-variant build
[kt-kernel] Loading avx2 from: /root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/kt_kernel/kt_kernel_ext.cpython-312-x86_64-linux-gnu.so
[kt-kernel] Successfully loaded AVX2 variant
[kt-kernel] Extension module loaded: kt_kernel_ext
[2026-05-10 06:47:28] Init torch distributed begin.
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[2026-05-10 06:47:28] Init torch distributed ends. elapsed=0.18 s, mem usage=0.06 GB
[2026-05-10 06:47:28] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4: No module named 'tilelang'
[2026-05-10 06:47:28] In import_model_classes: Ignore import error when loading sglang.srt.models.deepseek_v4_nextn: No module named 'tilelang'
[2026-05-10 06:47:28] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr: No module named 'transformers.models.glm_ocr'
[2026-05-10 06:47:28] In import_model_classes: Ignore import error when loading sglang.srt.models.glm_ocr_nextn: No module named 'transformers.models.glm_ocr'
[2026-05-10 06:47:28] In import_model_classes: Ignore import error when loading sglang.srt.models.glmasr: cannot import name 'GlmAsrConfig' from 'transformers' (/root/anaconda3/envs/fp-sglang/lib/python3.12/site-packages/transformers/__init__.py)
[2026-05-10 06:47:29] Load weight begin. avail mem=22.94 GB
[2026-05-10 06:47:29] Detected fp8 checkpoint.
[2026-05-10 06:47:29] Multimodal attention backend not set. Use triton_attn.
[2026-05-10 06:47:29] Using triton_attn as multimodal attention backend.
`torch_dtype` is deprecated! Use `dtype` instead!
[2026-05-10 06:47:29] Using kt_num_gpu_experts=2 per layer, total GPU experts: 120 (= 2 × 60 MoE layers)
[2026-05-10 06:47:29] Using uniform strategy for GPU expert placement
[2026-05-10 06:47:29] KT GPU experts: layer 0 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 1 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 2 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 3 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 4 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 5 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 6 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 7 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 8 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 9 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 10 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 11 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 12 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 13 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 14 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 15 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 16 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 17 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 18 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 19 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 20 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 21 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 22 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 23 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 24 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 25 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 26 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 27 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 28 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 29 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 30 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 31 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 32 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 33 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 34 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 35 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 36 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 37 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 38 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 39 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 40 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 41 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 42 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 43 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 44 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 45 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 46 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 47 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 48 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 49 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 50 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 51 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 52 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 53 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 54 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 55 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 56 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 57 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 58 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] KT GPU experts: layer 59 (MoE) has 2 GPU experts
[2026-05-10 06:47:29] Generated KT GPU experts masks using 'uniform' strategy: 60 MoE layers (out of 60 total layers) x 512 experts, total GPU experts in MoE layers = 120
[2026-05-10 06:47:29] [KT] Created shared staging buffer: 32.0 MiB (shape=torch.Size([4096, 4096]), dtype=torch.bfloat16)
CPUInfer[0x5623da609060]: Hello
WorkerPool[0x5623da608f30] 2 subpools, [numa:threads][0:36] [1:36]
===========In NumaPool============
In Numa Worker Pool at NUMA 0, 36 threads
===========In NumaPool============
In Numa Worker Pool at NUMA 1, 36 threads
[FP8SafeTensorLoader] Detected format: deepseek
[FP8SafeTensorLoader] Detected scale format: block-wise (weight_scale_inv)
[FP8SafeTensorLoader] Detected VL model
[2026-05-10 06:47:31] using attn output gate!
Loading safetensors checkpoint shards: 0% Completed | 0/94 [00:00

Inspect sglang/srt/layers/moe/fused_moe_triton/layer.py at _load_w13, then trace the FP8SafeTensorLoader path that reports a DeepSeek format and detects a VL model. Reproduce with the provided TP=1 command and verify that Qwen3.5-397B-A17B-FP8 loads without the 2048-versus-4096 shape mismatch across the reported settings.

Written by the indexing model from the issue text.