microsoft / microsoft/winml-cli
Add end-to-end document-question-answering inference and eval contract
@DingmaomaoBJTU is already working on this.
Since Jul 27, 2026.
- Dominant language
- Python
- Stars
- 40
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 50
Description
Summary
winml eval does not support the exact document-question-answering task contract. LayoutLM v1 document-QA models expose an extractive span head (input_ids, OCR-derived bbox, attention_mask, token_type_ids → start_logits, end_logits), but end-to-end task evaluation also requires OCR, word/box normalization and token alignment, answer-span decoding, and a document-QA dataset/metric adapter.
Current behavior
winml eval --schema --task document-question-answering
Error: Unsupported task 'document-question-answering'.
The generic inference task registry describes image + question inputs, but checkpoints such as DmitrySpartak/layoutlm-invoices have a tokenizer only and consume pre-tokenized OCR/layout tensors; they do not accept a PDF or image directly.
Desired behavior
Define an explicit document-QA inference/evaluation contract that:
- preserves
document-question-answeringrather than silently substituting plainquestion-answering; - supplies or integrates OCR and normalized bounding-box preprocessing;
- aligns question/document tokens and boxes;
- decodes
start_logits/end_logitsto answer spans; - provides a representative dataset and task metric schema;
- handles architecture differences such as LayoutLMv1 (text + boxes) versus LayoutLMv3 (text + boxes + pixels).
Scope note
This is separate from ONNX exporter registration tracked by #134. Export/build/perf can work on prepared tensors without providing an end-to-end document image-to-answer pipeline.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.