Can the pinned vLLM serve our Qwen3.8-27B Q4_K_M GGUF through vllm-gguf-plugin, and displace llama.cpp as this arm's oracle?
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: -
AGENTS.md §"When vLLM has no implementation" admits llama.cpp only where
vLLM implements nothing. The Qwen3.8-27B Q4_K_M GGUF arm is currently gated
against llama.cpp b10451 and that gate reads FAIL (#2534). If the pinned vLLM
can read the same GGUF through the first-party
vllm-gguf-plugin, then
llama.cpp is not the admissible oracle for this arm and the gate is pointed at
the wrong denominator.
vLLM's own docs/features/quantization/gguf.md at our pin 5559679229 names
that plugin as where in-tree GGUF support migrated. It is in the vllm-project
organisation, so it is not a secondary oracle; it is the primary oracle's own
out-of-tree extension.
The question this issue owns
Can the pinned vLLM, plus vllm-gguf-plugin, load and generate from
/mnt/nas_share/checkpoints/qwen3.8-27b-gguf/Qwen3.8-27B-Q4_K_M.gguf
(17,106,775,008 bytes, sha256 7e78da5d…, unsloth/Qwen3.8-27B-GGUF @
fe1e2a23d973adb629709749dc4f6756df66ef10) on a fleet device, and emit real
tokens? AGENTS.md §"When vLLM has no implementation" makes that the
gateability bar: constructing a config proves nothing.
Scope
- Source feasibility of the plugin on aarch64 + CUDA, cited to
file:line. - A build and a generation run inside an
rclease ondgx:gpu0. - An oracle record under
.agents/oracles/with a recorded pin, and the
argument for whether the plugin is part of thevllmoracle or a new
registry id —AGENTS.md's oracle table does not list it today, so adding a
row is a rule change that needs its own argument.
Explicitly out of scope
- Any speed, latency or memory number, and any cross-engine ratio. §Gates
forbids a performance result while the arm's token gate has not passed. - Changing the existing llama.cpp gate's verdict or its evidence documents.
- Advancing the vLLM parity pin.
Related
- #2534 — the failing Q4_K_M token gate against llama.cpp
b10451. - #2497 — the blocked quant-matched
gfx1151benchmark that cites the same
llama.cpp denominator. - #1003 — the owed llama.cpp floor re-take.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with AGENTS.md §"When vLLM has no implementation" and docs/features/quantization/gguf.md at pin 5559679229, then assess the vllm-gguf-plugin source for aarch64 and CUDA feasibility. Build and run generation against the specified GGUF in an rc lease on dgx:gpu0, confirming real tokens. Record the pin and result under .agents/oracles/ and explain whether it belongs to the vllm oracle or needs a new registry row.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ai-infra-agents, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100