Gemma-4 E2B multimodal (ORT-GenAI) eval vs HF PyTorch: quality-parity findings + per_layer_inputs release note

## Summary

We benchmarked the **Gemma-4 E2B-it multimodal (vision + audio) ORT-GenAI packages** against the HuggingFace PyTorch reference to validate export/runtime quality for the `fp16` and `Q4_K_M` (INT4 K-quant) variants. Headline: **the ORT-GenAI fp16 packages reach parity with HF PyTorch within eval noise** on all four benchmarks once a harness-side truncation artifact is accounted for; INT4 K-quant shows small, expected degradation concentrated on audio.

This issue documents the setup, results, and two notes relevant to the onnxruntime-genai team (one actionable: gemma-4 `per_layer_inputs` is only in `main`).

## Environment

| Component | Version |
|---|---|
| onnxruntime-genai-cuda | **0.15.0.dev0** (built from source, `main` @ `3c4b66e`) |
| CUDA | 13.0 (nvcc via `conda install -c nvidia/label/cuda-13.0.0 cuda-toolkit`) |
| GPU | NVIDIA H200 (arch sm_90) |
| transformers | 5.12.1 |
| torch | 2.12.1 |
| olive-ai | 0.11.0.dev0 (`LMMSEvaluator`, [Olive PR #2531](https://github.com/microsoft/Olive/pull/2531)) |
| lmms-eval | 0.7.2 ([DelwinKim fork, branch `t-delwinkim/hf-multimodal-fixes`](https://github.com/DelwinKim/lmms-eval/tree/t-delwinkim/hf-multimodal-fixes)) |

**Models:** `justinchuby/gemma-4-E2B-it-ONNX`, `cuda/` packages for `f16` and `Q4_K_M` (decoder + embedding + vision_encoder + audio_encoder + genai_config.json). HF reference: `google/gemma-4-E2B-it`.

## Note 1 (actionable): gemma-4 requires onnxruntime-genai `main`

Released onnxruntime-genai (≤ 0.14.1) **cannot load** the gemma-4 packages — `og.Model(...)` fails with:

```
Unknown value per_layer_inputs
```

`per_layer_inputs` (the gemma3n-style per-layer-embeddings input) is only supported on `main`. We had to build the CUDA wheel from source to run any evaluation:

```bash
git clone https://github.com/microsoft/onnxruntime-genai /tmp/ogenai && cd /tmp/ogenai # main @ 3c4b66e
CUDAARCHS=90 python build.py --use_cuda --cuda_home $CONDA_PREFIX --config Release --parallel --skip_tests
pip install --force-reinstall --no-deps build/Linux/Release/wheel/onnxruntime_genai_cuda-0.15.0.dev0-*.whl
```

**Ask:** cut a release (or document minimum version) that includes gemma-4 `per_layer_inputs` support so downstream users don't need a source build.

## Eval harness setup

Olive [`LMMSEvaluator` (PR #2531)](https://github.com/microsoft/Olive/pull/2531) drives lmms-eval on both the ORT-GenAI packages (via the `ortgenai_mm` adapter) and the HF PyTorch model. Tasks (full sets), greedy decoding:

- `ocrbench` (accuracy, 1000 samples)
- `docvqa_val_lite` (ANLS)
- `chartqa_lite` (relaxed accuracy)
- `librispeech_test_clean` (WER, ↓ better)

HF is vision-only: lmms-eval's generic `huggingface` chat wrapper can't feed audio (`Unsupported visual type: AudioDecoder`), so LibriSpeech is ORT-GenAI-only. gemma-4 also needs `model_class: "huggingface"` in the evaluator config (not in Olive's auto-map).

## Results

All numbers are full sets. OCRBench values use `ignore_stop_strings: ["\n\n"]` (see Note 2); the other three tasks are unaffected by that flag (bit-identical with/without).

| Benchmark (metric) | HF PyTorch | ORT-GenAI fp16 | ORT-GenAI K-Quant (Q4_K_M) |
|---|---|---|---|
| **OCRBench** (acc ↑) | 0.727 | **0.712** | 0.699 |
| **DocVQA** val_lite (ANLS ↑) | 0.737 | **0.745** | 0.710 |
| **ChartQA** lite (relaxed ↑) | 0.416 | **0.426** | 0.426 |
| **LibriSpeech** test-clean (WER ↓) | — | **5.55** | 6.81 |

### Finding A — fp16 export fidelity is within noise
HF → ORT-GenAI fp16: OCRBench −0.015, DocVQA **+0.008**, ChartQA **+0.010**. The fp16 packages match (and on 2/3 vision tasks slightly exceed) HF. On OCRBench specifically the two models **agree on 955/1000** samples; the 45 disagreements are near-symmetric (HF-only-correct 30, genai-only-correct 15).

Precision is **not** the cause of the residual: forcing HF to `float16` gives the **identical** 0.727 as its default bf16 (22/1000 predictions changed, 0 score change) — the model is numerically robust to fp16 vs bf16 at this task level. The remaining ~1.5 pt OCRBench delta is (i) a substring-match scoring artifact that rewards HF's more verbose output, and (ii) symmetric greedy-tie flips on genuinely ambiguous fine-text (`4950`↔`4990`, `1998`↔`1999`), i.e. PyTorch-bf16 vs ORT-fp16 kernels resolving the argmax differently on a coin-flip. Not a systematic quality loss.

### Note 2 — benchmarking caveat: ORT-GenAI generation is more concise; watch harness stop-strings
lmms-eval injects a default `until=["\n\n"]` (the fewshot delimiter) for tasks that don't set their own. The `ortgenai_mm` adapter honors it; the HF wrapper effectively does not. Gemma-4 answers in two paragraphs (`preamble\n\n**answer**`), so the ORT-GenAI side was **truncated at the first blank line**, cutting off the part containing the answer. Because OCRBench scores by substring-match (`answer in output`), this cost the genai packages ~9 points that had nothing to do with model quality:

| | OCRBench fp16 | OCRBench K-Quant |
|---|---|---|
| default (`until=["\n\n"]`) | 0.640 | 0.611 |
| `ignore_stop_strings: ["\n\n"]` | **0.712** | **0.699** |

Diagnostic: 0/98 of the initially-disagreeing genai OCRBench outputs contained a blank line (they all stopped at the first `\n\n`), while HF's ground-truth string appeared *after* the first blank line in 59 of them. This is an Olive/lmms-eval harness interaction, not an onnxruntime-genai bug — flagged here as guidance for anyone benchmarking genai multimodal packages with substring/anls-style metrics.

### Finding B — quantization (fp16 → Q4_K_M) impact
Vision QA degrades modestly (OCRBench −0.013, DocVQA −0.035, ChartQA 0). **Audio degrades most: WER 5.55 → 6.81 (~+23% relative).** ASR is the most quantization-sensitive of these tasks.

**What Q4_K_M actually quantizes — decoder only.** Comparing the two packages byte-for-byte, only the LLM decoder is quantized; the audio encoder, vision encoder, and embedding table are left in fp16 and are **bit-identical** across variants:

| Component (`model.onnx.data`) | f16 | Q4_K_M | Quantized? |
|---|---|---|---|
| audio_encoder | 614,727,680 | 614,727,680 | No — identical fp16 |
| vision_encoder | 358,088,704 | 358,088,704 | No — identical fp16 |
| embedding | 5,530,517,504 | 5,530,517,504 | No — identical fp16 |
| **decoder (LLM)** | 4,733,337,600 | **1,511,784,448** | **Yes — INT4 (~0.32×)** |

So the audio WER regression is **not** an encoder-quantization effect (the audio encoder is untouched) — it comes entirely from the **INT4-quantized decoder**. ASR is hit harder than vision QA because WER penalizes every wrong/missing word across a long autoregressive transcription, so small INT4 logit perturbations accumulate; the vision QA metrics (substring / ANLS) tolerate paraphrase and barely move under the same decoder quant.

**Levers to recover audio quality are on the decoder, not the encoders:** ship a higher-precision decoder for ASR use (fp16, or a gentler quant such as Q8 / int4 with a higher-precision K-quant config), use mixed precision keeping the most sensitive decoder layers higher, or recommend the fp16 package for audio-critical scenarios and reserve Q4_K_M for vision/text-QA where the hit is ≤3.5 pts.

## Reproduction (per run)

```bash
# ORT-GenAI runtime needs CUDA libs on LD_LIBRARY_PATH (CUDA 13 toolkit + cu13 pip libs)
export LD_LIBRARY_PATH=$CONDA_PREFIX/lib:$CONDA_PREFIX/lib/python3.12/site-packages/nvidia/cu13/lib:/usr/local/cuda/lib64:$LD_LIBRARY_PATH
```

Evaluator config (ONNX side); set `ignore_stop_strings: ["\n\n"]` for OCRBench-style multi-paragraph tasks:

```json
{
"input_model": { "type": "OnnxModel", "model_path": ".../f16/cuda/decoder", "onnx_file_name": "model.onnx" },
"evaluators": { "e": {
"type": "LMMSEvaluator",
"tasks": ["ocrbench", "docvqa_val_lite", "chartqa_lite", "librispeech_test_clean"],
"batch_size": 1, "max_new_tokens": 256, "max_length": 4096,
"ignore_stop_strings": ["\n\n"]
}},
"evaluator": "e", "evaluate_input_model": true, "target": "local"
}
```

HF reference run adds `"model_class": "huggingface"` and drops the audio task.

## TL;DR for the onnxruntime-genai team
1. **fp16 gemma-4 E2B packages are quality-validated** — parity with HF PyTorch within eval noise on OCRBench/DocVQA/ChartQA.
2. **Please release / document a version with gemma-4 `per_layer_inputs` support** (currently `main`-only; released ≤0.14.1 errors `Unknown value per_layer_inputs`).
3. Benchmarking note: genai generation is more concise than HF; substring/ANLS metrics + a harness `until=["\n\n"]` will under-score it — suppress that stop for multi-paragraph tasks.