mudler / mudler/vllm.cpp

Target-only Qwen3.8 W4 sampled decode selects mixed-dtype packed GDN

Open
#1,842 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Native Windows CUDA target-only decode for the Qwen3.8-27B W4A16 gate model fails before producing a sampled token when temperature is 0.7 and top-p is 0.8. The production path reaches vt::GdnPackedDecode with mixed_qkv/a/b/out not sharing one FP16/BF16/F32 dtype and throws at src/vt/ops.cpp:2496. The same target model runs through speculative DFlash2 verification because that step takes a different GDN recurrence route. Reproduction evidence: J:\vllmcpp-dflash2-w4-runs\73-target-only-repro-t07-p08-1x4. Owning row: SPEC-REJECTION. The fix must preserve the packed-decode dtype predictor/producer invariant and add a target-only production regression test.\n\nFOLLOWING_AGENTS_PROTOCOL\n\nFollowing-Agents-Protocol: true\nAI-Assisted: true\nAssisted-by: AGENT:GPT-5 [CODEX]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the mixed-dtype failure in src/vt/ops.cpp:2496 and use the reproduction evidence at J:\vllmcpp-dflash2-w4-runs\73-target-only-repro-t07-p08-1x4 to trace the target-only production path into vt::GdnPackedDecode. Compare the packed-decode dtype predictor and producer, then add a target-only production regression test covering the Qwen3.8-27B W4A16 sampled decode case. Done means the path preserves the shared dtype invariant and produces a sampled token without the rejection.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.