antirez / antirez/ds4

DSpark preserved-MXFP4 planner support and mixed IQ2/MXFP4 Metal tests

Offen
#642 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
C
Sterne
22.4k
Forks
2.1k
Ø Merge
1 T. 3 Std.
Gemergte PRs (30 T.)
4

Beschreibung

> **Disclosure:** This issue was prepared and filed by **Codex (OpenAI)** on behalf of `@apetersson` while attempting to build a 128 GB-optimized, selectively native-MXFP4 DeepSeek V4 Flash quant on the experimental `ds4f-mxfp4` branch. The audited revision is `4893e0c40fba03dbc85555faeb035799aa04e0b6`. Please independently review the analysis and proposed patch.

## Summary

The branch can plan and generate preserved native MXFP4 routed experts for the main GGUF, but the equivalent DSpark support plan rejects `--routed-w2 mxfp4` before generation:

```text
loaded imatrix ...: 138 entries
error: unsupported DSpark planned tensor type
```

This blocks a mixed DSpark support model with IQ2_XXS gate/up and losslessly preserved native MXFP4 down experts. The generator and current Metal runtime appear to have the necessary primitives, but the intended mixed 1–5-token DSpark path also lacks targeted coverage.

This is related to, but narrower than, #641: that issue describes type-39 runtime support on older branches. This report is about the new branch's DSpark converter/planner and mixed-path tests.

## Reproduction

Using a native DeepSeek V4 Flash checkpoint whose routed experts are packed I8 weights plus F8_E8M0 scales:

```sh
./gguf-tools/deepseek4-quantize \
--hf \
--template \
--dspark-support \
--experts iq2_xxs \
--routed-w2 mxfp4 \
--imatrix \
--imatrix-strict \
--dspark-block-size 5 \
--dspark-target-layers 40,41,42 \
--out /tmp/dspark-mxfp4.gguf \
--dry-run
```

Result:

```text
error: unsupported DSpark planned tensor type
```

## Apparent cause

The main planner has a deliberately narrow preserved-expert exception:

```c
const bool preserved_mxfp4 =
type == DS4Q_TYPE_MXFP4 && parse_expert_tensor(src->name).is_expert;
```

`dspark_plan_set_size()` instead rejects any type that is not I32 or an ordinary quantizable target:

```c
if (tp->meta.type != DS4Q_TYPE_I32 && !is_quantizable_target(tp->meta.type)) {
die("unsupported DSpark planned tensor type");
}
```

MXFP4 is intentionally not an ordinary requantization target, so making it generically quantizable would be the wrong fix. The exception should remain constrained to compatible packed routed experts.

The DSpark generator already recognizes `mtp.*` expert tensors and reaches the common native MXFP4 repacker after planning.

## Proposed converter change

Mirror the main planner's preserved-MXFP4 exception in the DSpark planner, with all of the following constraints:

- The tensor is a DSpark routed expert tensor.
- The source weight is packed I8 and the paired scale is F8_E8M0.
- Shape and scale shape match the native repacker contract.
- The logical first dimension is MXFP4-block aligned.
- MXFP4 remains rejected for ordinary/non-expert tensors.

## Missing Metal test coverage

The planned support layout is:

- Gate: IQ2_XXS
- Up: IQ2_XXS
- Down: MXFP4
- DSpark proposal block: up to 5 tokens

Please add a CPU/reference versus Metal test covering token counts `1, 2, 3, 4, 5`, including:

- The fused tiny gate/up path.
- MXFP4 down plus fused expert sum for 1–4 rows.
- The generic MXFP4 MV plus expert-summation path used by the 5-row block.
- A real support-binding/verifier smoke test if a small fixture can cover it.

The current all-MXFP4 Metal test is useful but does not exercise this mixed DSpark layout.

## Why this matters for the 128 GB profile

For the three DSpark down aggregates:

- Q4_K sidecar: approximately `6.9840 GiB`
- Native-MXFP4-down sidecar: approximately `6.7965 GiB`
- Saving: exactly `0.1875 GiB`

The larger benefit is fidelity: the three down tensors retain the checkpoint's exact FP4 codes and scale bytes instead of being requantized. The main model remains authoritative, so this primarily affects proposal fidelity, acceptance and speed rather than final target-model correctness.

## Acceptance criteria

- Strict dry-run succeeds with `--dspark-support --routed-w2 mxfp4`.
- Exactly the three intended DSpark down aggregate tensors are MXFP4.
- Non-expert MXFP4 requests still fail.
- Native codes and scale bytes are checked during generation.
- Mixed IQ2/IQ2/MXFP4 Metal tests pass for 1–5 tokens on Apple Silicon.
- A real DSpark verifier smoke test completes without binding, bounds or correctness errors.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start at dspark_plan_set_size() and the existing main-planner preserved_mxfp4 exception; compare the DSpark generator's mtp.* handling and the current all-MXFP4 Metal test. Reproduce the strict dry-run, then exercise CPU/reference versus Metal for 1–5 tokens, checking the three down aggregates' native codes and scale bytes and the non-expert rejection. Finish with the DSpark verifier smoke test if the fixture supports it.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
c
Bereich
machine-learning, testing-qa
Issue-Typ
Feature
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
45/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.