mudler / mudler/vllm.cpp

EXL3 decoders take a const void* weight operand, so a wrongly typed argument no longer fails to compile

Open
#2,597 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: -

Owed by .agents/specs/unaligned-safetensors-consumers.md under ## Owed.

What changed, and what it cost

#2581 repaired four CPU consumers that formed a const uint16_t* over a borrowed safetensors payload at an odd address. Fixing the EXL3 pair meant the trellis cursor became a byte cursor, so five signatures widened their weight-side operand from const uint16_t* to const void*:

Signature Declared in
vt::Exl3TileCodeword(const void* tile, ...) include/vt/ops.h
vt::Exl3DecodeTile(const void* tile, ...) include/vt/ops.h
vt::Exl3ReconstructInner(const void* trellis, ...) include/vt/ops.h
vt::Exl3DequantLinear(const void* trellis, const void* suh, const void* svh, ...) include/vt/ops.h
MoeGemm(const uint16_t* a_had, const void* trellis, ...) src/vt/cpu/cpu_exl3_kernels.cpp

HadRowBlock(HadIo, const void* in, void* out, const void* pre, const void* post, ...) in the same file widened its pre and post scale operands for the same reason.

Why this is a hazard and not a defect

const void* accepts ANY pointer type without a diagnostic. Before the change, passing an f32 scale array, a byte length where a word count belongs, or an unrelated tensor to Exl3DequantLinear was a compile error. Now it compiles and decodes garbage. The stride argument that used to be supplied by the pointer type is carried explicitly instead, so a caller that forgets the factor of two reads the wrong half of every word and still type-checks.

Nothing is wrong TODAY. Every existing caller passes the int16 words as stored, a uint16_t* converts implicitly, and no call site outside the repaired files changed. test_exl3_gemm, test_qwen35_exl3 and test_exl3_dequant are green under address,undefined, and #2581's new case compares the odd-address decode BYTE-FOR-BYTE against the even-address one, so a lost factor of two reds it. The exposure is to the NEXT caller, which has no compiler telling it what the parameter means.

Fix shape

A one-member wrapper -- struct Exl3Trellis { const void* words; } or an equivalent vt::ByteSpan -- restores the diagnostic without restoring the alignment requirement, because the wrapper's own alignment is 1. It must NOT be a re-typed const uint16_t*: that is the undefined load #2558 and #2578 name, and reverting to it reintroduces the abort under -fsanitize=alignment.

The change is mechanical but it crosses the public ABI in include/vt/ops.h, so it wants its own row, its own spec, and the EXL3 suites green beside it rather than riding in a fix for a different defect. Filed rather than swept.

Found during the fresh review of #2581 (finding 2, rated non-blocking).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .agents/specs/unaligned-safetensors-consumers.md and the EXL3 declarations in include/vt/ops.h, then inspect src/vt/cpu/cpu_exl3_kernels.cpp and its callers. Introduce the diagnostic-preserving wrapper described in the issue without restoring typed unaligned loads, update the affected call sites, and run test_exl3_gemm, test_qwen35_exl3, and test_exl3_dequant under address and undefined sanitizers.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend-api-design, machine-learning, performance, testing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
64/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.