bert / fill-mask + feature-extraction + document-question-answering: all models pass wmk perf

Summary

The bert family has 4 failing models across different root causes: two multilingual models fail due to Windows HF cache symlink issues, one model is an ONNX-only repository with no PyTorch weights, and one model fails because document-question-answering is not supported by TasksManager for bert. The goal is for all bert models to pass wmk perf.

Eval Results (2026-03-11)

Failing (4 models):

Status	Model	Task	Error	Root Cause
FAIL	google-bert/bert-base-multilingual-uncased	fill-mask	Model not found: model.safetensors (exit 3)	Windows HF cache symlink broken
FAIL	google-bert/bert-base-multilingual-cased	fill-mask	same (exit 3)	same
FAIL	Xenova/paraphrase-multilingual-MiniLM-L12-v2	feature-extraction	does not appear to have pytorch_model.bin / model.safetensors (exit 4)	ONNX-only repo — no PyTorch weights
FAIL	pardeepSF/layoutlm-vqa	document-question-answering	Task 'document-question-answering' not supported by TasksManager (exit 4)	bert model_type but doc-qa task not registered

Passing (representative):

Status	Model	Task
PASS	google-bert/bert-base-uncased	fill-mask
PASS	google-bert/bert-large-uncased-whole-word-masking-finetuned-squad	question-answering
PASS	google-bert/bert-base-multilingual-cased	feature-extraction
PASS	google-bert/bert-base-multilingual-cased	masked-lm
PASS	Intel/bert-base-uncased-mrpc	feature-extraction
PASS	Intel/bert-base-uncased-mrpc	text-classification
PASS	dslim/bert-base-NER	token-classification
PASS	dbmdz/bert-large-cased-finetuned-conll03-english	token-classification
PASS	deepset/bert-large-uncased-whole-word-masking-squad2	question-answering

Root Cause Analysis

1. Google-bert multilingual (exit code 3 — symlink)

HuggingFace Hub uses symlinks to deduplicate cached files. On Windows without Developer Mode, symlinks are not supported — the cache directory is created but model.safetensors is a broken symlink. The HF SDK logs a warning for all models but only these two fail to resolve the file, suggesting they were downloaded during a session where symlink creation silently failed.

2. Xenova/paraphrase-multilingual-MiniLM-L12-v2 (exit code 4 — ONNX-only)

The Xenova namespace publishes ONNX-only models (no PyTorch weights). ModelKit's HF loader requires .safetensors or pytorch_model.bin. This model cannot be loaded through the current pipeline.

3. pardeepSF/layoutlm-vqa (exit code 4 — doc-qa task)

This model has model_type=bert in config but is fine-tuned for document-question-answering using LayoutLM-style inputs. The task document-question-answering is not registered in Optimum's TasksManager for the bert model type. Note: the LayoutLM/LayoutLMv3 issue (#441) tracks the same root cause for layoutlm-native model types.

Desired State

All bert family models pass wmk perf.

Acceptance Criteria

• google-bert/bert-base-multilingual-uncased / fill-mask passes wmk perf
• google-bert/bert-base-multilingual-cased / fill-mask passes wmk perf
• Xenova/paraphrase-multilingual-MiniLM-L12-v2 / feature-extraction passes wmk perf (requires ONNX-first loading path or clear unsupported error)
• pardeepSF/layoutlm-vqa / document-question-answering passes wmk perf
• All currently passing bert models continue to pass
• Fix is universal — no hardcoded model names (CLAUDE.md Cardinal Rule #1)
• uv run pytest tests/ passes (CLAUDE.md Cardinal Rule #3)

Technical Notes

Fix for multilingual bert (symlink):

• Clear corrupted HF cache entries and re-download with Developer Mode enabled, OR
• Set HF_HUB_DISABLE_SYMLINKS_WARNING=1 and ensure local_files_only=False forces re-download as regular files
• ModelKit should detect exit code 3 (ModelNotFoundError) caused by broken symlinks and emit a clear recovery message: "HF cache may be corrupted. Try: huggingface-cli delete-cache for this model and re-download."

Fix for Xenova ONNX-only (no PyTorch weights):

• Option A: Add an ONNX-first loading path in modelkit/loader/hf.py — detect that the repo has .onnx files but no .safetensors, and skip the PyTorch export step
• Option B: Emit a clear UnsupportedModelError with message "This model only provides ONNX weights. Use wmk perf --onnx <path> instead."

Fix for pardeepSF/layoutlm-vqa (doc-qa task for bert):

• Register document-question-answering as a supported task for bert model type, or
• Detect the task mismatch at loader time and reroute to the correct Auto class
• Coordinate with #441 (layoutlm doc-qa) — same TasksManager registration gap

Related Files

• modelkit/loader/hf.py — HF model loading, symlink error handling
• modelkit/loader/config.py — task resolution and model class detection
• modelkit/export/io.py — register_onnx_overwrite() for doc-qa task registration
• eval_results/2026-03-11/models/google-bert__bert-base-multilingual-uncased__fill-mask/result.json
• eval_results/2026-03-11/models/Xenova__paraphrase-multilingual-MiniLM-L12-v2__feature-extraction/result.json
• eval_results/2026-03-11/models/pardeepSF__layoutlm-vqa__document-question-answering/result.json

References

• Related: #441 (layoutlm doc-qa task — same TasksManager gap)
• CLAUDE.md Cardinal Rule #1: No hardcoded model names
• CLAUDE.md Cardinal Rule #3: Run uv run pytest tests/ after implementation