Whisper: hidden_size > 0 assertion in lm_config.hpp:121 crashes flm serve --asr 1 and flm list after Whisper pull

Summary

On Linux FLM 0.9.41, every documented invocation that loads whisper-v3:turbo crashes with an unconditional assert(this->hidden_size > 0) in LM_Config::from_pretrained. Whisper's config.json does not (and should not) have a hidden_size field — it has d_model, encoder_layers, etc. — so the assertion fires whenever this code path is reached for a Whisper config. The assertion is still present in trunk (src/include/lm_config.hpp:121), so this affects current main too.

Additional side effect: once flm pull whisper-v3:turbo succeeds, every subsequent flm list invocation crashes at exit because the listing code iterates all installed config.json files and hits the assertion on Whisper.

Reproducer (all three paths produce identical crash)

$ flm pull whisper-v3:turbo                # succeeds; downloads 4 files (~622 MB) into ~/.config/flm/models/Whisper-V3-Turbo-NPU2/

$ flm serve --asr 1 --port 11434           # docs: "standalone ASR model" mode
[FLM]  Using custom model list path: /opt/fastflowlm/share/flm/model_list.json
[FLM]  ASR mode enabled: reserving additional 1GB of memory
[FLM]  Using user-specified port: 11434
flm: src/include/lm_config.hpp:121:
void LM_Config::from_pretrained(std::string): Assertion `this->hidden_size > 0' failed.
Aborted (core dumped)

$ flm serve llama3.2:3b --asr 1 --port 11434  # docs: sidecar pattern
[same assertion crash, same line]

$ flm list                                  # iterates installed configs at exit
[FLM]  Using custom model list path: /opt/fastflowlm/share/flm/model_list.json
Models:
  - deepseek-r1-0528:8b ✅
  - llama3.2:1b ✅
  - llama3.2:3b ✅
  - qwen3:0.6b ✅
  - qwen3:4b ✅
  - qwen3:8b ✅
  - qwen3-it:4b ✅
  - qwen3.5:4b ✅
  - qwen3.5:9b ✅
flm: lm_config.hpp:121: void LM_Config::from_pretrained(std::string): Assertion `this->hidden_size > 0' failed.
Aborted

The flm list output is complete before the abort (so it's user-visible), but the non-zero exit code breaks any scripting that checks flm list's return status.

Expected behavior

flm serve --asr 1 should start the Whisper ASR server cleanly (per the docs "standalone ASR model" example). flm list should not crash regardless of whether Whisper is installed.

Mechanism

whisper-v3:turbo's config.json (downloaded by flm pull) has:

{
  "_name_or_path": "/raid/yoach/tmp_whisper_turbo",
  "architectures": ["WhisperForConditionalGeneration"],
  "model_type": "whisper",
  "d_model": 1280,
  "encoder_layers": 32,
  "encoder_attention_heads": 20,
  "decoder_layers": 4,
  ...
  "flm_version": "0.9.14"
}

No hidden_size field — Whisper's encoder/decoder transformer uses d_model for its width, not the chat-LLM hidden_size terminology.

LM_Config::from_pretrained in src/include/lm_config.hpp (line ~121 in 0.9.41, still present in current main) unconditionally asserts:

assert(this->vocab_size > 0);
assert(this->hidden_size > 0);          // ← crashes on Whisper
assert(this->intermediate_size > 0);
assert(this->num_attention_heads > 0);
assert(this->num_hidden_layers > 0);
assert(this->num_key_value_heads > 0);

The same class has fields for audio_model_weight and audio_config, which suggests the design intent is "audio support is embedded inside the chat model's config.json" — but the actual --asr 1 codepath ends up loading Whisper's own config and tripping the assertion regardless.

Regression bisection: introduced in v0.9.39

The assert(this->hidden_size > 0) itself isn't new — it's been in
lm_config.hpp since at least v0.9.14 (verified by fetching the file
at v0.9.14, v0.9.22, v0.9.30, v0.9.36, v0.9.37, v0.9.38, v0.9.39, v0.9.40,
v0.9.41 — all contain the assertion). So Whisper used to take a code
path that didn't go through LM_Config::from_pretrained.

What changed: commit e2ccfd5d ("feat: update config for audio", 2026-04-07) added audio-config handling to LM_Config itself:

+   std::string audio_model_weight;
+   nlohmann::json _audio_config;
+   bool is_audio;

…and added the corresponding JSON loads inside the from_pretrained
body, right next to the existing hidden_size > 0 assertion.

This landed in v0.9.39 (released 2026-04-15). v0.9.38 (2026-04-02)
predates it.

Cross-checks:

• Issue #441 (closed 2026-03-27) confirms Whisper-via-FLM worked then
  on the same Ryzen AI 9 HX 370 chip — that user was likely on
  v0.9.36 or v0.9.37, both predating the regression.
• FLM's own benchmark page fastflowlm.com/docs/benchmarks/qwen3.5_results/
  cites measurements taken on v0.9.38 — i.e. the last known-working
  release for the audio loader.

Regression range: introduced between v0.9.38 (works) and v0.9.39 (broken).

Likely mechanism: before v0.9.39, the --asr 1 codepath either
bypassed LM_Config::from_pretrained for Whisper, or chained the
chat-LLM config as the "main" config (which has hidden_size). The
v0.9.39 refactor routed both standalone (flm serve --asr 1) and
sidecar (flm serve <chat> --asr 1) Whisper loads through
LM_Config::from_pretrained, where the unconditional hidden_size > 0
assert has always been waiting.

Suggested fix

from_pretrained should branch on model_type: when model_type == "whisper", validate Whisper-specific fields (d_model, encoder_layers, encoder_attention_heads, etc.) instead of the chat-LLM ones. Or the chat-LLM assertions should be gated to apply only when the config represents a chat LLM.

A minimal patch that avoids breaking existing behavior: relax the asserts to early-return-with-error when model_type == "whisper", deferring to the dedicated Whisper loader path (whichever currently exists for the ASR/-a flag).

Confirmed not a Linux-packaging artifact

This was triaged on Gentoo Linux. ldd /opt/fastflowlm/bin/flm confirms the binary IS linked against the full audio stack at runtime:

libavformat.so.62, libavcodec.so.62, libswresample.so.6, libswscale.so.9,
libavutil.so.60, libfftw3f.so.3, libboost_program_options.so.1.90.0,
libreadline.so.8, libcurl.so.4, and /opt/fastflowlm/lib64/flm/libwhisper_npu.so

So the Whisper compute path is compiled in; the crash is in FLM's own config-parsing code, not in missing native deps.

Environment

• Hardware: Framework 13, AMD Ryzen AI 9 HX 370 (Strix Point), XDNA 2 NPU,
• NPU FW: 1.1.2.64
• amdxdna: 1.0 (kernel 7.0.2)
• OS: Gentoo Linux
• FLM: 0.9.41 (built from upstream tag via Gentoo sci-ml/fastflowlm ebuild)
• XRT / xrt-xdna: 2.21.75

flm validate output:

[FLM]  Using custom model list path: /opt/fastflowlm/share/flm/model_list.json
[Linux]  Kernel: 7.0.2
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 1.0
[Linux]  Memlock Limit: infinity

Cross-references

• Docs page that documents the --asr 1 standalone pattern: fastflowlm.com/docs/models/whisper/
• Model registry entry: models.whisper-v3.turbo in src/model_list.json (flm_min_version: "0.9.14")
• Trunk source: src/include/lm_config.hpp line ~121 (assertion is present)

Start by reading src/include/lm_config.hpp around line 121 and reproducing the failure with flm list and flm serve --asr 1. Trace the Whisper model_type path and audio-config handling added in commit e2ccfd5d; done means Whisper loads without an assertion and flm list exits successfully while existing chat-model validation remains intact.