mudler / mudler/vllm.cpp

perf(cpu): add packed Cortex-A76 Q8 multi-output kernel

Open
#623 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Track KERNEL-CPU-A76-Q8-DOT follow-up work to close the measured Raspberry Pi 5 gap against llama.cpp by increasing data reuse rather than rescheduling the existing nrc=1 leaf.

Scope:

  • profile and speed-limit model the current A76 Q8_0 x Q8_0 path;
  • add one explicit, non-default packed 1x4 or 4x4 candidate through the shared MatmulBTQuant dispatch;
  • preserve strict block-order floating-point accumulation and exact output/token gates;
  • prove selector, packing/layout, tails, alignment, ABI and reporting with RED-first tests and mutations;
  • require pinned GCC13 AArch64 disassembly plus QEMU correctness before any Pi measurement;
  • compare same-binary current A76 assembly vs candidate on M=1 and reached prefill shapes with wall time, cycles, instructions, frontend/backend stalls and cache traffic;
  • do not change the automatic default unless physical Pi and recursive model gates pass every throughput/latency/memory axis.

The current one-output assembly measures about 14.94 cycles per two Q8 blocks versus an optimistic ~10-cycle SIMD-port bound, while the full prefill profile attributes ~30% of cycles to Q8 and shows backend/cache rather than frontend pressure. The candidate hypothesis is activation reuse across four output rows, following the project-pinned llama.cpp 1x4/4x4 structure but adapting it to this project strict reduction-order contract.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the shared MatmulBTQuant dispatch and profile the current A76 Q8_0 x Q8_0 path on the pinned GCC13 AArch64 build. Add the non-default packed 1x4 or 4x4 candidate, then run RED-first selector, packing/layout, tail, alignment, ABI, mutation, and QEMU correctness checks. Done requires pinned disassembly and same-binary M=1/reached-prefill comparisons; do not change defaults until Pi and recursive model gates pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, raspberry-pi
Domain
backend, performance, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.