mudler / mudler/vllm.cpp

Twenty more const uint16_t* casts over borrowed safetensors bytes remain after the six that landed

Open
#2,579 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: -

Owed by .agents/specs/unaligned-safetensors-consumers.md under ## Owed.

What was measured

Repairing the three consumers named in #2540 and #2558 exposed a fourth of the same shape on the very next sanitizer run (#2578, WeightF32), because -fno-sanitize-recover=all stops the binary at the first report and each repair only reveals the next site.

A grep over src/ at aedad724c finds 20 further occurrences of the identical shape -- a const uint16_t* formed over an OwnedTensor's borrowed bytes:

grep -rn "reinterpret_cast<const uint16_t\*>([A-Za-z_.]*bytes\.data())" src/
File Sites
models/gemma4_moe.cpp 10
models/gemma4_weights.cpp 2
models/dots3_note_device.cpp 1
models/nemotron_h_device.cpp 1
models/nemotron_h.cpp 1
models/deepseek_v2_weights.cpp 1
models/glm_moe_dsa_loader.cpp 1
models/ltx2_loader.cpp 1
models/minicpm3_weights.cpp 1
models/kimi_linear_weights.cpp 1

What was NOT measured, and why this is filed rather than fixed

Whether each of those sites is actually reached with an ODD base is unmeasured. BorrowStTensorBytes only borrows when direct upload is on and the tensor is a whole-range view, and a checkpoint whose header length happens to be even puts every tensor on an even address. So this list is a population of the SHAPE, not a list of confirmed findings, and turning it into findings needs a run per model rather than a grep.

Fixing 20 sites across ten model files is also not the unit of work #2540 and #2558 describe, and each one wants the model's own suite green beside it.

Fix shape

Identical to the six that landed: a byte cursor plus vt::LoadUnaligned<uint16_t> (include/vt/unaligned.h), which at -O2 compiles to the same load. Do NOT gate the borrow instead: .agents/specs/unaligned-safetensors-consumers.md records the measurement showing a producer-side alignment refusal switches direct upload off for every tensor in roughly half of all checkpoints.

A checker that refuses this shape outright would be the durable answer, and #627 already proposed one for the loader half of the class.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .agents/specs/unaligned-safetensors-consumers.md and run the provided grep over src/ to review the 20 sites in the ten listed model files. Check each consumer's model-specific suite and sanitizer coverage before applying the established unaligned-load approach. Done means all listed borrowed-byte casts are addressed and the relevant model suites remain green.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.