microsoft / microsoft/winml-cli
[Task] object-detection EP coverage investigation
@ssss141414 is already working on this.
Since Apr 9, 2026.
- Dominant language
- Python
- Stars
- 40
- Forks
- 11
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 50
Description
Overview
Object detection models predict bounding boxes and class labels for objects in images. This task covers table structure detection (table-transformer), real-time detection (RT-DETR), and YOLO-style transformers (YOLOS), plus fashion-domain detection. All 10 models pass on at least one EP.
This issue focuses on the 5 models with incomplete EP coverage — primarily a VitisAI failure pattern (4 models), plus one model failing on both QNN and VitisAI. Root cause investigation needed across DETR and RT-DETR architectures.
Agent Scenarios
- Document intelligence agent: extract table structure from scanned documents or PDFs using table-transformer models, enabling structured data extraction from unstructured layouts
- Retail / inventory agent: detect product locations in warehouse or shelf images (yolos-fashionpedia, RT-DETR) for automated inventory tracking
- Visual copilot agent: ground natural language object references to bounding boxes in a scene, enabling spatial reasoning for multimodal agents
EP Coverage Status
| Model | QNN | OV | VitisAI |
|---|---|---|---|
| microsoft/table-transformer-detection | PASS | PASS | FAIL |
| microsoft/table-transformer-structure-recognition-v1.1-all | PASS | PASS | FAIL |
| PekingU/rtdetr_r101vd_coco_o365 | PASS | PASS | FAIL |
| PekingU/rtdetr_v2_r18vd | PASS | PASS | FAIL |
| PekingU/rtdetr_r50vd_coco_o365 | FAIL | PASS | FAIL |
Reference: microsoft/table-transformer-structure-recognition, hustvl/yolos-small, facebook/detr-resnet-50, valentinafeve/yolos-fashionpedia, TahaDouaji/detr-doc-table-detection already pass all 3 EPs.
Acceptance Criteria
- microsoft/table-transformer-detection passes all 3 EPs (investigate VitisAI failure)
- microsoft/table-transformer-structure-recognition-v1.1-all passes all 3 EPs (investigate VitisAI failure)
- PekingU/rtdetr_r101vd_coco_o365 passes all 3 EPs (investigate VitisAI failure)
- PekingU/rtdetr_v2_r18vd passes all 3 EPs (investigate VitisAI failure)
- PekingU/rtdetr_r50vd_coco_o365 passes all 3 EPs (investigate QNN + VitisAI failure)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.