Alberto-Codes / Alberto-Codes/vramfit

MoE target: fit Qwen3-Coder-30B-A3B on a 24 GiB card above stock 4-bit quality

Open
#548 2 comments 0 reactions 0 assignees View on GitHub
target
Dominant language
Python
Stars
1
Forks
1
Avg merge
6h 8m
Merged PRs (30d)
105

Description

## Destination

Pack **Qwen3-Coder-30B-A3B-Instruct** so that one 24 GiB consumer card
serves it at a quality the stock 4-bit build does not reach.

The destination is a measured mixed-precision artifact that:

- loads fully offloaded on one 24 GiB card at the ruled context
- generates
- stays inside a weight budget measured for this architecture, not inherited
- carries PPL ratio and mean KLD against the reference in one instrument frame
- beats the stock `Q4_K_M` build on both metrics

This is a **dogfood target**. The maintainer runs this model daily on the
reference box, so a win is one he uses rather than one he reads about.

## Why the target is out of reach today

The published shelf on the Ollama library lists `30b-a3b-q8_0` at **32GB**
and `30b-a3b-q4_K_M` at **19GB** (https://ollama.com/library/qwen3-coder/tags).

The repo's 24 GiB reference weight budgets at 16k context are **18.94 GiB**
with fp16 KV and **20.47 GiB** with fp8 KV
(`docs/explanation/vram-budget.md:167`). A 32GB-class file is far above both.

The reference box measured `24564` MiB total and `22031` MiB free on
2026-09-05, so the free envelope is narrower than the nominal card.

So the 8-bit build is genuinely out of reach and the 4-bit build is what the
card gets today. The gap between them is the whole target.

**Caveat a worker must not skip.** The 18.94 / 20.47 GiB figures are the
worked example for the Nemotron 49B attention shape at 16k. They are
reference points, not this target's budget. Recompute with
`vramfit budget --model-config --vram 24GiB
--context --kv-dtype fp8` before any recipe is solved. If
the recomputed budget disagrees with the reference figures, the recomputed
budget binds and this issue gets the correction as a comment.

## Why the shape suits vramfit

Qwen3-Coder-30B-A3B carries about 30.5B total parameters with 3.3B active
(https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct). The shape is
memory-heavy and compute-light. Most of the mass sits in routed expert
stacks that vramfit can compress unevenly while the backbone every token
traverses keeps precision.

Chart #158 already proved that pattern on the same class of shape: the
nineteenth data point beat the published build and served under the cap
(`docs/explanation/evaluating-packed-models.md:1937`).

So this is a transfer test of decisions the 30B campaign already earned. It
is not a blank-sheet MoE project.

## Comparators

- **Fit baseline:** stock `30b-a3b-q4_K_M`. The maintainer's local copy
measured `18,550,716,416` bytes at `--ctx-size 32768`.
- **No-fit quality ceiling:** stock `30b-a3b-q8_0`, about 32GB.

A vramfit recipe that lands between them in bytes and above `Q4_K_M` in
measured quality is the result this target wants.

## The bar

A result is interesting when all of these hold:

- the artifact loads fully offloaded on one 24 GiB card at the ruled context
- the stock `Q8_0` build remains far outside the card
- the stock `Q4_K_M` build fits but has materially worse measured damage
- the vramfit recipe wins on **both** PPL ratio and mean KLD, per #380

## Stop rule

If byte arithmetic from the real converted tensor surface shows that the
non-expert block plus the cheapest acceptable expert palette cannot fit the
measured budget, close this target before any paid scan.

If the pack surface needs a class map this repo does not carry, raise that
compatibility gap as its own issue rather than growing this one.

Do not let this issue become a general Qwen runtime project.

## Out of scope

Which model the maintainer adopts as his daily driver, and which serving
path he standardizes on, are open maintainer decisions. This ticket stands
either way. It is a packing target, not an adoption ruling.

## Sequence

The measurement that proves or kills this target is its own ticket.
This issue closes on an artifact and a scoreboard row, or on a measured
reason the target cannot reach the destination.

## Source facts

- Ollama `qwen3-coder` tag sizes: https://ollama.com/library/qwen3-coder/tags
- Qwen/Qwen3-Coder-30B-A3B-Instruct model card
- `docs/explanation/vram-budget.md:167` — 24 GiB reference weight budgets at 16k
- `docs/explanation/evaluating-packed-models.md:1937` — chart #158 nineteenth data point

Contributor guide

Open the contributing guide

Research direction

Start with the Qwen3-Coder config and run the stated `vramfit budget` command at the ruled context and fp8 KV setting; the recomputed budget binds. Read `docs/explanation/vram-budget.md:167` and `docs/explanation/evaluating-packed-models.md:1937`, then inspect the converted tensor surface before any scan. Done means an artifact and scoreboard row meeting the offload and quality bars, or a measured reason the target cannot fit.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, machine-learning, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.