mudler / mudler/vllm.cpp

Can the pinned vLLM serve our Qwen3.8-27B Q4_K_M GGUF through vllm-gguf-plugin, and displace llama.cpp as this arm's oracle?

Open
#2,624 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: -

AGENTS.md §"When vLLM has no implementation" admits llama.cpp only where
vLLM implements nothing
. The Qwen3.8-27B Q4_K_M GGUF arm is currently gated
against llama.cpp b10451 and that gate reads FAIL (#2534). If the pinned vLLM
can read the same GGUF through the first-party
vllm-gguf-plugin, then
llama.cpp is not the admissible oracle for this arm and the gate is pointed at
the wrong denominator.

vLLM's own docs/features/quantization/gguf.md at our pin 5559679229 names
that plugin as where in-tree GGUF support migrated. It is in the vllm-project
organisation, so it is not a secondary oracle; it is the primary oracle's own
out-of-tree extension.

The question this issue owns

Can the pinned vLLM, plus vllm-gguf-plugin, load and generate from
/mnt/nas_share/checkpoints/qwen3.8-27b-gguf/Qwen3.8-27B-Q4_K_M.gguf
(17,106,775,008 bytes, sha256 7e78da5d…, unsloth/Qwen3.8-27B-GGUF @
fe1e2a23d973adb629709749dc4f6756df66ef10) on a fleet device, and emit real
tokens? AGENTS.md §"When vLLM has no implementation" makes that the
gateability bar: constructing a config proves nothing.

Scope

  • Source feasibility of the plugin on aarch64 + CUDA, cited to file:line.
  • A build and a generation run inside an rc lease on dgx:gpu0.
  • An oracle record under .agents/oracles/ with a recorded pin, and the
    argument for whether the plugin is part of the vllm oracle or a new
    registry id — AGENTS.md's oracle table does not list it today, so adding a
    row is a rule change that needs its own argument.

Explicitly out of scope

  • Any speed, latency or memory number, and any cross-engine ratio. §Gates
    forbids a performance result while the arm's token gate has not passed.
  • Changing the existing llama.cpp gate's verdict or its evidence documents.
  • Advancing the vLLM parity pin.

Related

  • #2534 — the failing Q4_K_M token gate against llama.cpp b10451.
  • #2497 — the blocked quant-matched gfx1151 benchmark that cites the same
    llama.cpp denominator.
  • #1003 — the owed llama.cpp floor re-take.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with AGENTS.md §"When vLLM has no implementation" and docs/features/quantization/gguf.md at pin 5559679229, then assess the vllm-gguf-plugin source for aarch64 and CUDA feasibility. Build and run generation against the specified GGUF in an rc lease on dgx:gpu0, confirming real tokens. Record the pin and result under .agents/oracles/ and explain whether it belongs to the vllm oracle or needs a new registry row.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai-infra-agents, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.