Alberto-Codes / Alberto-Codes/vramfit
MoE target: fit Qwen3-Coder-30B-A3B on a 24 GiB card above stock 4-bit quality
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- Avg merge
- 6h 8m
- Merged PRs (30d)
- 105
Description
## Destination
Pack **Qwen3-Coder-30B-A3B-Instruct** so that one 24 GiB consumer card
serves it at a quality the stock 4-bit build does not reach.
The destination is a measured mixed-precision artifact that:
- loads fully offloaded on one 24 GiB card at the ruled context
- generates
- stays inside a weight budget measured for this architecture, not inherited
- carries PPL ratio and mean KLD against the reference in one instrument frame
- beats the stock `Q4_K_M` build on both metrics
This is a **dogfood target**. The maintainer runs this model daily on the
reference box, so a win is one he uses rather than one he reads about.
## Why the target is out of reach today
The published shelf on the Ollama library lists `30b-a3b-q8_0` at **32GB**
and `30b-a3b-q4_K_M` at **19GB** (https://ollama.com/library/qwen3-coder/tags).
The repo's 24 GiB reference weight budgets at 16k context are **18.94 GiB**
with fp16 KV and **20.47 GiB** with fp8 KV
(`docs/explanation/vram-budget.md:167`). A 32GB-class file is far above both.
The reference box measured `24564` MiB total and `22031` MiB free on
2026-09-05, so the free envelope is narrower than the nominal card.
So the 8-bit build is genuinely out of reach and the 4-bit build is what the
card gets today. The gap between them is the whole target.
**Caveat a worker must not skip.** The 18.94 / 20.47 GiB figures are the
worked example for the Nemotron 49B attention shape at 16k. They are
reference points, not this target's budget. Recompute with
`vramfit budget --model-config --vram 24GiB
--context --kv-dtype fp8` before any recipe is solved. If
the recomputed budget disagrees with the reference figures, the recomputed
budget binds and this issue gets the correction as a comment.
## Why the shape suits vramfit
Qwen3-Coder-30B-A3B carries about 30.5B total parameters with 3.3B active
(https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct). The shape is
memory-heavy and compute-light. Most of the mass sits in routed expert
stacks that vramfit can compress unevenly while the backbone every token
traverses keeps precision.
Chart #158 already proved that pattern on the same class of shape: the
nineteenth data point beat the published build and served under the cap
(`docs/explanation/evaluating-packed-models.md:1937`).
So this is a transfer test of decisions the 30B campaign already earned. It
is not a blank-sheet MoE project.
## Comparators
- **Fit baseline:** stock `30b-a3b-q4_K_M`. The maintainer's local copy
measured `18,550,716,416` bytes at `--ctx-size 32768`.
- **No-fit quality ceiling:** stock `30b-a3b-q8_0`, about 32GB.
A vramfit recipe that lands between them in bytes and above `Q4_K_M` in
measured quality is the result this target wants.
## The bar
A result is interesting when all of these hold:
- the artifact loads fully offloaded on one 24 GiB card at the ruled context
- the stock `Q8_0` build remains far outside the card
- the stock `Q4_K_M` build fits but has materially worse measured damage
- the vramfit recipe wins on **both** PPL ratio and mean KLD, per #380
## Stop rule
If byte arithmetic from the real converted tensor surface shows that the
non-expert block plus the cheapest acceptable expert palette cannot fit the
measured budget, close this target before any paid scan.
If the pack surface needs a class map this repo does not carry, raise that
compatibility gap as its own issue rather than growing this one.
Do not let this issue become a general Qwen runtime project.
## Out of scope
Which model the maintainer adopts as his daily driver, and which serving
path he standardizes on, are open maintainer decisions. This ticket stands
either way. It is a packing target, not an adoption ruling.
## Sequence
The measurement that proves or kills this target is its own ticket.
This issue closes on an artifact and a scoreboard row, or on a measured
reason the target cannot reach the destination.
## Source facts
- Ollama `qwen3-coder` tag sizes: https://ollama.com/library/qwen3-coder/tags
- Qwen/Qwen3-Coder-30B-A3B-Instruct model card
- `docs/explanation/vram-budget.md:167` — 24 GiB reference weight budgets at 16k
- `docs/explanation/evaluating-packed-models.md:1937` — chart #158 nineteenth data point
Contributor guide
Research direction
Start with the Qwen3-Coder config and run the stated `vramfit budget` command at the ruled context and fp8 KV setting; the recomputed budget binds. Read `docs/explanation/vram-budget.md:167` and `docs/explanation/evaluating-packed-models.md:1937`, then inspect the converted tensor surface before any scan. Done means an artifact and scoreboard row meeting the offload and quality bars, or a measured reason the target cannot fit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100