microsoft / microsoft/winml-cli

Add end-to-end document-question-answering inference and eval contract

Open
#1,132 2 comments 0 reactions 1 assignee View on GitHub

@DingmaomaoBJTU is already working on this.

Since Jul 27, 2026.

enhancement model / task scale P2 triaged
Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
50

Description

Summary

winml eval does not support the exact document-question-answering task contract. LayoutLM v1 document-QA models expose an extractive span head (input_ids, OCR-derived bbox, attention_mask, token_type_idsstart_logits, end_logits), but end-to-end task evaluation also requires OCR, word/box normalization and token alignment, answer-span decoding, and a document-QA dataset/metric adapter.

Current behavior

winml eval --schema --task document-question-answering
Error: Unsupported task 'document-question-answering'.

The generic inference task registry describes image + question inputs, but checkpoints such as DmitrySpartak/layoutlm-invoices have a tokenizer only and consume pre-tokenized OCR/layout tensors; they do not accept a PDF or image directly.

Desired behavior

Define an explicit document-QA inference/evaluation contract that:

  • preserves document-question-answering rather than silently substituting plain question-answering;
  • supplies or integrates OCR and normalized bounding-box preprocessing;
  • aligns question/document tokens and boxes;
  • decodes start_logits / end_logits to answer spans;
  • provides a representative dataset and task metric schema;
  • handles architecture differences such as LayoutLMv1 (text + boxes) versus LayoutLMv3 (text + boxes + pixels).

Scope note

This is separate from ONNX exporter registration tracked by #134. Export/build/perf can work on prepared tensors without providing an end-to-end document image-to-answer pipeline.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.