intel / intel/torch-xpu-ops

[OOB][Perf] aten::cat performance gap on XPU vs CUDA (54 models affected)

Open
#3,780 0 comments 0 reactions 1 assignee Claimed by @jianyizh View on GitHub
test: oob
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

## Summary

`aten::cat` is the top 1 op priority for XPU performance improvement in OOB 300 eager inference (fp16). Improving cat to match CUDA roofline efficiency would give the largest fleet geomean R improvement.

## Data (from roofline profiling, 154 models, fp16 inference)

| Platform | Geomean R Delta | Models Affected | Total Saving (ms) |
|----------|------:|------:|------:|
| B580 | +0.0147 | 54 | 8,717 |
| G31 (B70) | +0.0082 | 46 | 1,033 |

Current fleet geomean R: B580=0.644, G31=0.675, 4080S=0.683.

## Worst affected models

| Model | BS | Suite | B580 R | 4080S R | G31 R | B580/4080S T2 | G31/4080S T2 | Top Gap Op |
|-------|---:|-------|------:|------:|------:|------:|------:|------|
| hf_Reformer | 128 | torchbench | 0.167 | 0.735 | 0.555 | 7.60x | 1.75x | aten::cat |
| dpn107 | 512 | timm | 0.646 | 0.746 | 0.615 | 1.87x | 1.45x | aten::cat |
| res2net50_14w_8s | 1024 | timm | 0.569 | 0.673 | 0.589 | 2.06x | 1.49x | aten::cat |
| yolov3 | 128 | torchbench | 0.865 | 0.790 | 0.697 | 1.39x | 1.22x | aten::cat |
| moondream | 1 | torchbench | 0.585 | 0.687 | 0.503 | 1.35x | 1.01x | aten::cat |
| selecsls42b | 1024 | timm | 0.682 | 0.752 | 0.674 | 1.71x | 1.25x | aten::cat |
| BartForCausalLM | 32 | huggingface | 0.778 | 0.835 | 0.725 | 1.27x | 0.89x | aten::cat |
| MBartForCausalLM | 32 | huggingface | 0.779 | 0.831 | 0.725 | 0.83x | 0.58x | aten::cat |

## Methodology

T1/T2/R roofline projection framework. R_op = projected_time / actual_time per op. If R_op on XPU < R_op on CUDA for the same model, the XPU kernel is underperforming relative to hardware capability. "Total Saving" = sum across all affected models of (actual_xpu - target_xpu) where target = proj / R_cuda.

## Hardware

- B580: 93 TFLOPS fp16, 410 GB/s
- G31 (B70): 154 TFLOPS fp16, 532 GB/s
- 4080S: 100.96 TFLOPS fp16, 716.8 GB/s

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.