microsoft / microsoft/winml-cli
bart + mbart / summarization + text-classification + zero-shot-classification: all models pass wmk perf
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 40
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 50
Description
Summary
All BART-family models fail across summarization, text-classification, and zero-shot-classification tasks. Two distinct errors: (1) EncoderDecoderCache JIT type error during seq2seq export, and (2) index -1 is out of bounds for dimension 1 with size 0 during classification inference.
Eval Results (2026-03-11)
| Status | Model | Task | Error |
|---|---|---|---|
| FAIL | facebook/bart-large-cnn | summarization | EncoderDecoderCache unsupported JIT input |
| FAIL | sshleifer/distilbart-cnn-12-6 | summarization | same |
| FAIL | philschmid/bart-large-cnn-samsum | summarization | same |
| FAIL | knkarthick/MEETING_SUMMARY | summarization | same |
| FAIL | moussaKam/mbarthez | summarization | same (mbart model_type) |
| FAIL | facebook/bart-large-mnli | text-classification | index -1 is out of bounds for dimension 1 with size 0 |
| FAIL | facebook/bart-large-mnli | zero-shot-classification | same |
| FAIL | joeddav/bart-large-mnli-yahoo-answers | zero-shot-classification | same |
| FAIL | valhalla/distilbart-mnli-12-3 | zero-shot-classification | same |
9/9 models fail — 0 pass.
Root Cause
Two separate issues:
- Seq2seq (summarization): Same
EncoderDecoderCacheTorchScript incompatibility as T5 family — see T5 issue. - Classification (text-classification / zero-shot-classification): BART uses a decoder with a special EOS token as the classification token. The input generation produces an empty decoder input, causing
index -1 is out of bounds for dimension 1 with size 0when the model tries to extract logits from the last token position.
Current State
- No
bart.pymodel config inmodelkit/models/hf/ - No ONNX config override registered for
bartmodel type - Relies on Optimum defaults which do not handle BART's classification head correctly
Desired State
All 9 BART/mBART models above pass wmk perf.
Acceptance Criteria
- All 9 models listed above pass
wmk perf - Fix is universal — no hardcoded model names (CLAUDE.md Cardinal Rule #1)
-
uv run pytest tests/passes (CLAUDE.md Cardinal Rule #3) - Eval re-run confirms 9/9 pass
Technical Notes
- For seq2seq: Same fix as T5 —
use_cache=Falseduring export or customBartOnnxConfig - For classification: BART's
BartForSequenceClassificationusesdecoder_input_idsstarting with EOS. The dummy input generator must provide validdecoder_input_ids. Register aBartIOConfigvia@register_onnx_overwrite("bart", "text-classification", ...)with correct decoder input shape. eval_results/2026-03-11/models/facebook__bart-large-mnli__text-classification/result.jsonshows the ONNX export succeeds, but inference fails — meaning the ONNX graph is valid but the input shape is wrong at perf time.- Consider a unified encoder-decoder ONNX config base class for bart/t5/marian
Related Files
modelkit/models/hf/bert.py— pattern for custom ONNX configmodelkit/export/io.py—register_onnx_overwrite()modelkit/models/hf/__init__.py:47-54— add bart toMODEL_BUILD_CONFIGSeval_results/2026-03-11/models/facebook__bart-large-mnli__text-classification/result.jsoneval_results/2026-03-11/models/google-t5__t5-small__summarization/result.json
References
- CLAUDE.md Cardinal Rule #1: No hardcoded model type strings
- CLAUDE.md Cardinal Rule #3: Run
uv run pytest tests/after implementation
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with modelkit/models/hf/bert.py and modelkit/export/io.py, then inspect modelkit/models/hf/init.py:47-54 and the referenced result.json files. Run the listed BART perf cases and the test suite to confirm the failures; done means all nine models pass wmk perf and uv run pytest tests/ passes without hardcoded model names.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100