antirez / antirez/ds4

test_deepseek41_metal: new router check fails on M1 Ultra (pre-M5); sqrt(softplus) compared at 3e-6 but 1.0f+exp is ~0.7% imprecise

Aperta
#1,039 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C
Stelle
22.4k
Fork
2.1k
Merge medio
1g 3h
PR unite (30g)
4

Descrizione

`tests/test_deepseek41_metal` fails the new router check on M1 Ultra:

```
router n=256 mode=0 rows=1 row=0 expert=4 logit=-11.8886719 actual=0.00263455603 ref=0.00262947031
tests/test_deepseek41_metal.c:105: fabs(probs[t * n + e] - ref[e]) <= 3e-6 * (1 + ref[e])
ds4: Metal cleanup discarded 5 live tensor handles
make: *** [test-deepseek41-metal] Error 1
```

## Repro

```sh
make tests/test_deepseek41_metal && ./tests/test_deepseek41_metal
```

No model or GGUF needed. Fails every run, identical values.

## Environment

| | |
|---|---|
| ds4 commit | `a04f46f` (DeepSeek v4.1 Flash support for CUDA) |
| Hardware | Mac Studio, Apple M1 Ultra, 128 GB |
| OS | macOS 26.6.2 (Darwin 25.6.0) |
| Backend | Metal, `make` default target |
| Metal 4 | unavailable, engine logs `Metal 4 tensor API disabled for pre-M5/pre-A19 devices` |

## Not a regression: the check is new

`check_router` is added wholesale by `a04f46f`; every line of it is new in that commit's diff of `tests/test_deepseek41_metal.c`. Confirmed by bisect on this machine:

| commit | result |
|---|---|
| `bd66c40` DeepSeek v4.1 Flash support for Metal | PASS |
| `a04f46f` DeepSeek v4.1 Flash support for CUDA | FAIL |

So the kernel behaviour is presumably unchanged; the new check is simply tighter than pre-M5 Apple GPUs can satisfy.

## Root cause

The check compares two independent FP32 evaluations of `sqrt(softplus(x))` for a logit of about -11.9, where that expression is ill-conditioned.

Kernel, `metal/dsv4_misc.metal:5188` (and the same form at `:5349`):

```metal
const float4 sp = select(log(1.0f + exp(x)), x, x > 20.0f);
```

Host reference, `tests/test_deepseek41_metal.c:98`:

```c
ref[e] = sqrtf(v > 20 ? (float)v : logf(1.0f + expf((float)v)));
```

For `x = -11.8886719`, `exp(x)` is about 6.87e-6. Adding that to `1.0f` discards most of it: float resolution near 1.0 is 1.19e-7, so the addend carries roughly **0.68% relative error** before the log. Against the true value:

| | value | error vs exact |
|---|---|---|
| exact, `sqrt(log1p(exp(x)))` in double | 0.00262063751 | — |
| host reference, `1.0f + expf` | 0.00262947031 | +8.83e-6 |
| Metal kernel | 0.00263455603 | +1.39e-5 |
| FP32 using `log1pf(expf(x))` | 0.00262063742 | -9.17e-11 |

Both sides are wrong by several times the tolerance, in the same direction, by different amounts. Their gap is 5.09e-6 against a bound of 3.01e-6. The check therefore only passes where the GPU happens to round this expression the same way the host does, which is presumably true on M5 Max and M3 Ultra and is not true here.

Note the second row of that table independently: the router probability the kernel produces is about **0.53% high** for logits near -12. Switching the kernel to `log1p(exp(x))` would remove essentially all of it.

## Suggested fix

Two options, and the choice is yours since they differ in intent:

1. **Fix the reference only.** Use `log1pf(expf(v))` in the test and widen the tolerance enough to cover the kernel's genuine FP32 error at small logits. Keeps kernel behaviour, makes the check portable. The comment at `tests/test_deepseek41_metal.c:96` says the `1.0f +` form was chosen deliberately to match Metal, so this is the smaller change.
2. **Fix the kernel too.** Use `log1p(exp(x))` in `metal/dsv4_misc.metal:5188` and `:5349`, then keep a tight tolerance against an accurate reference. This also removes the 0.53% routing-probability error above.

## Scope of verification

Verified on Metal, M1 Ultra only. CUDA and ROCm untested. I do not have an M5 or M3 to confirm the check passes there.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.