intel / intel/graph-compiler

bf16 matmul's corresponding `tensor.pack` not properly optimized

Open
#320 4 comments 1 reaction 1 assignee Claimed by @BRUCE11111 View on GitHub
performance
Dominant language
C++
Stars
54
Forks
16
PR merge metrics
No merged PRs in 30d

Description

Currently, the following 2 single-layer MLP have worst performance compared with GC v1.

<!--table
{mso-displayed-decimal-separator:"\.";
mso-displayed-thousand-separator:"\,";}
@page
{margin:.75in .7in .75in .7in;
mso-header-margin:.3in;
mso-footer-margin:.3in;}
tr
{mso-height-source:auto;}
col
{mso-width-source:auto;}
br
{mso-data-placement:same-cell;}
td
{padding-top:1px;
padding-right:1px;
padding-left:1px;
mso-ignore:padding;
color:black;
font-size:11.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:"Aptos Narrow", sans-serif;
mso-font-charset:0;
mso-number-format:General;
text-align:general;
vertical-align:bottom;
border:none;
mso-background-source:auto;
mso-pattern:auto;
mso-protection:locked visible;
white-space:nowrap;
mso-rotate:0;}
.xl65
{mso-number-format:"0\.0000";}
.xl66
{color:red;
mso-number-format:Percent;}
.xl67
{color:red;
mso-number-format:"0\.0000";}
-->

dtype | batch size | hidden list | GC V1 | 8c55a0544 remove brgemm read lock
-- | -- | -- | -- | --
bf16 | 128 | 1024x1024 | 0.0286 | 0.0828 | 34.52%
bf16 | 128 | 1024x512 | 0.0204 | 0.0670 | 30.45%

We performed detailed breakdown as follows:

<!--table
{mso-displayed-decimal-separator:"\.";
mso-displayed-thousand-separator:"\,";}
@page
{margin:.75in .7in .75in .7in;
mso-header-margin:.3in;
mso-footer-margin:.3in;}
tr
{mso-height-source:auto;}
col
{mso-width-source:auto;}
br
{mso-data-placement:same-cell;}
td
{padding-top:1px;
padding-right:1px;
padding-left:1px;
mso-ignore:padding;
color:black;
font-size:11.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:"Aptos Narrow", sans-serif;
mso-font-charset:0;
mso-number-format:General;
text-align:general;
vertical-align:bottom;
border:none;
mso-background-source:auto;
mso-pattern:auto;
mso-protection:locked visible;
white-space:nowrap;
mso-rotate:0;}
-->

128x1024x1024 | GC v1 | 8c55a0544
-- | -- | --
matmul only | 0.01766 | 0.01989
tiled pack (or reorder) | 0.02634 | 0.04632
total | 0.04418 | 0.077969

and

<!--table
{mso-displayed-decimal-separator:"\.";
mso-displayed-thousand-separator:"\,";}
@page
{margin:.75in .7in .75in .7in;
mso-header-margin:.3in;
mso-footer-margin:.3in;}
tr
{mso-height-source:auto;}
col
{mso-width-source:auto;}
br
{mso-data-placement:same-cell;}
td
{padding-top:1px;
padding-right:1px;
padding-left:1px;
mso-ignore:padding;
color:black;
font-size:11.0pt;
font-weight:400;
font-style:normal;
text-decoration:none;
font-family:"Aptos Narrow", sans-serif;
mso-font-charset:0;
mso-number-format:General;
text-align:general;
vertical-align:bottom;
border:none;
mso-background-source:auto;
mso-pattern:auto;
mso-protection:locked visible;
white-space:nowrap;
mso-rotate:0;}
.xl65
{color:windowtext;}
-->

128x1024x512 | GC v1 | 8c55a0544
-- | -- | --
matmul only | 0.01587 | 0.01591
tiled pack (or reorder) | 0.01278 | 0.0398
total | 0.02881 | 0.06917

Are there any further optimization opportunity for vnni pack?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.