mudler / mudler/vllm.cpp

--fit sizes a placement from ON-DISK bytes, so it can leave a tower on a device whose keep-quant set cannot hold it

Open
#2,565 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: ENG-HYBRID-PLACEMENT

--fit (ResolveMoeFitFromSizes, src/vllm/model_executor/device_placement.cpp)
decides how many trailing layers to place from GgufRoutedExpertBytesPerLayer,
which sums t.nbytes -- the ON-DISK size of each routed-expert tower. It has no
term for the RESIDENCY those towers will take on the engine device, and on a
device whose keep-quant set does not cover the file's expert encodings an
UNPLACED tower does not cost its on-disk size. It expands, or the load refuses.

Measured on strix:gpu0 (gfx1151, ROCm 7.2.4), GLM-5.3 UD-IQ1_S,
vllm-cli --device auto with no placement configured (so --fit is on by
default), on a build of #2562:

engine: device placement INSTALLED: 56 layers run their routed experts on cpu,
  the rest on rocm (resolved against 78 layers, origin fit)
engine: device placement: --fit placed 56 layer(s) (147798884352 B) to bring a
  216433205760 B footprint under a 68719476736 B budget; WHOLE layers only
vllm-cli: model load failed (status 2): vllm_engine_load: vt: glm-dsa gguf:
  blk.3.ffn_gate_exps.weight routed to an EXPAND residency ...

The fit check itself now passes -- #2562 credits the plan, which is #2517 -- so
this is no longer masked by an earlier refusal. blk.3 is one of the 22 layers
--fit left on the device, its ffn_gate_exps is IQ1_S, and
DeviceKeepQuantSupported serves {Q8_0, Q4_K, Q5_K, Q6_K} on ROCm, so it
expands and LoadStackedExperts refuses by name.

cpu_moe (VT_CPU_MOE=1), which places all 78, loads this checkpoint on this
board today. So the fix is not urgent for the operator who knows; it is a default
that produces a refusal on a configuration nobody chose.

What a fix owes

A decision about whether the fit resolver may consult the residency policy at
all. It runs BEFORE the policy is built today (InstallMoePlacementPlan at
model_loader.cpp:2672, GgufLoadPolicy::FromEnv inside each registry), and
inverting that order is a real change to the load sequence rather than a
tweak -- which is why this is filed rather than folded into #2562.

Note also the interaction: an unplaced IQ1_S tower's EXPANDED size is several
times its on-disk size, so a residency-aware resolver would place strictly more
layers than the current one, not fewer.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ResolveMoeFitFromSizes in src/vllm/model_executor/device_placement.cpp, then trace InstallMoePlacementPlan at model_loader.cpp:2672 and GgufLoadPolicy::FromEnv in each registry. Determine how fit can account for the residency policy without breaking the load sequence. Done means the GLM-5.3 UD-IQ1_S case no longer leaves an expanding tower on the device or fails during LoadStackedExperts.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.