microsoft / microsoft/winml-cli

[Task] object-detection EP coverage investigation

Open
#296 0 comments 0 reactions 1 assignee View on GitHub

@ssss141414 is already working on this.

Since Apr 9, 2026.

model / task scale P2 triaged
Dominant language
Python
Stars
40
Forks
11
Avg merge
1d 8h
Merged PRs (30d)
50

Description

Overview

Object detection models predict bounding boxes and class labels for objects in images. This task covers table structure detection (table-transformer), real-time detection (RT-DETR), and YOLO-style transformers (YOLOS), plus fashion-domain detection. All 10 models pass on at least one EP.

This issue focuses on the 5 models with incomplete EP coverage — primarily a VitisAI failure pattern (4 models), plus one model failing on both QNN and VitisAI. Root cause investigation needed across DETR and RT-DETR architectures.

Agent Scenarios

  • Document intelligence agent: extract table structure from scanned documents or PDFs using table-transformer models, enabling structured data extraction from unstructured layouts
  • Retail / inventory agent: detect product locations in warehouse or shelf images (yolos-fashionpedia, RT-DETR) for automated inventory tracking
  • Visual copilot agent: ground natural language object references to bounding boxes in a scene, enabling spatial reasoning for multimodal agents

EP Coverage Status

Model QNN OV VitisAI
microsoft/table-transformer-detection PASS PASS FAIL
microsoft/table-transformer-structure-recognition-v1.1-all PASS PASS FAIL
PekingU/rtdetr_r101vd_coco_o365 PASS PASS FAIL
PekingU/rtdetr_v2_r18vd PASS PASS FAIL
PekingU/rtdetr_r50vd_coco_o365 FAIL PASS FAIL

Reference: microsoft/table-transformer-structure-recognition, hustvl/yolos-small, facebook/detr-resnet-50, valentinafeve/yolos-fashionpedia, TahaDouaji/detr-doc-table-detection already pass all 3 EPs.

Acceptance Criteria

  • microsoft/table-transformer-detection passes all 3 EPs (investigate VitisAI failure)
  • microsoft/table-transformer-structure-recognition-v1.1-all passes all 3 EPs (investigate VitisAI failure)
  • PekingU/rtdetr_r101vd_coco_o365 passes all 3 EPs (investigate VitisAI failure)
  • PekingU/rtdetr_v2_r18vd passes all 3 EPs (investigate VitisAI failure)
  • PekingU/rtdetr_r50vd_coco_o365 passes all 3 EPs (investigate QNN + VitisAI failure)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.