microsoft / microsoft/winml-cli

bart + mbart / summarization + text-classification + zero-shot-classification: all models pass wmk perf

Open
#120 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dev experience model / task scale NPU P2 triaged
Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
50

Description

Summary

All BART-family models fail across summarization, text-classification, and zero-shot-classification tasks. Two distinct errors: (1) EncoderDecoderCache JIT type error during seq2seq export, and (2) index -1 is out of bounds for dimension 1 with size 0 during classification inference.

Eval Results (2026-03-11)

Status Model Task Error
FAIL facebook/bart-large-cnn summarization EncoderDecoderCache unsupported JIT input
FAIL sshleifer/distilbart-cnn-12-6 summarization same
FAIL philschmid/bart-large-cnn-samsum summarization same
FAIL knkarthick/MEETING_SUMMARY summarization same
FAIL moussaKam/mbarthez summarization same (mbart model_type)
FAIL facebook/bart-large-mnli text-classification index -1 is out of bounds for dimension 1 with size 0
FAIL facebook/bart-large-mnli zero-shot-classification same
FAIL joeddav/bart-large-mnli-yahoo-answers zero-shot-classification same
FAIL valhalla/distilbart-mnli-12-3 zero-shot-classification same

9/9 models fail — 0 pass.

Root Cause

Two separate issues:

  1. Seq2seq (summarization): Same EncoderDecoderCache TorchScript incompatibility as T5 family — see T5 issue.
  2. Classification (text-classification / zero-shot-classification): BART uses a decoder with a special EOS token as the classification token. The input generation produces an empty decoder input, causing index -1 is out of bounds for dimension 1 with size 0 when the model tries to extract logits from the last token position.

Current State

  • No bart.py model config in modelkit/models/hf/
  • No ONNX config override registered for bart model type
  • Relies on Optimum defaults which do not handle BART's classification head correctly

Desired State

All 9 BART/mBART models above pass wmk perf.

Acceptance Criteria

  • All 9 models listed above pass wmk perf
  • Fix is universal — no hardcoded model names (CLAUDE.md Cardinal Rule #1)
  • uv run pytest tests/ passes (CLAUDE.md Cardinal Rule #3)
  • Eval re-run confirms 9/9 pass

Technical Notes

  • For seq2seq: Same fix as T5 — use_cache=False during export or custom BartOnnxConfig
  • For classification: BART's BartForSequenceClassification uses decoder_input_ids starting with EOS. The dummy input generator must provide valid decoder_input_ids. Register a BartIOConfig via @register_onnx_overwrite("bart", "text-classification", ...) with correct decoder input shape.
  • eval_results/2026-03-11/models/facebook__bart-large-mnli__text-classification/result.json shows the ONNX export succeeds, but inference fails — meaning the ONNX graph is valid but the input shape is wrong at perf time.
  • Consider a unified encoder-decoder ONNX config base class for bart/t5/marian

Related Files

  • modelkit/models/hf/bert.py — pattern for custom ONNX config
  • modelkit/export/io.pyregister_onnx_overwrite()
  • modelkit/models/hf/__init__.py:47-54 — add bart to MODEL_BUILD_CONFIGS
  • eval_results/2026-03-11/models/facebook__bart-large-mnli__text-classification/result.json
  • eval_results/2026-03-11/models/google-t5__t5-small__summarization/result.json

References

  • CLAUDE.md Cardinal Rule #1: No hardcoded model type strings
  • CLAUDE.md Cardinal Rule #3: Run uv run pytest tests/ after implementation

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with modelkit/models/hf/bert.py and modelkit/export/io.py, then inspect modelkit/models/hf/init.py:47-54 and the referenced result.json files. Run the listed BART perf cases and the test suite to confirm the failures; done means all nine models pass wmk perf and uv run pytest tests/ passes without hardcoded model names.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.