[RFC]Performance Optimization Skills Suite on Intel Platforms

🚀 The feature, motivation and pitch

Performance Optimization Skills Suite on Intel Platforms

Task List

• Kernel performance analysis: build the evidence chain based on the kernel implementation and Unitrace counter.
  #4306

• ASM code extract and source code mapping:
  #4176
  #4177

• Hardware workflow skill PRs to cover the EU analysis path:
  #4307 adds eu-utilization-triage, the ComputeBasic three-state entry workflow for routing stall / ILP / TLP follow-up analysis.
  #4308 adds eu-stall-attribution, the workflow for high XVE_STALL attribution using stall counters, per-IP data, ASM extraction, and ASM/source mapping.
  #4309 adds eu-ilp-coissue, the workflow for low co-issue / pipe imbalance analysis with ComputeBasic pipe counters and ASM evidence.
  #4310 adds eu-tlp-occupancy, the workflow for deciding when more thread-level parallelism helps versus worsens Stage-2 Pipe/Send contention.

---

1. Motivation

Given a PyTorch model or an operator, the goal of perfomrance optimization is to improve the hardware resource ultization to approach the roofline goal. This process need a lot of expertise. For example, when you want optmize the attention layer in the LLM modesl on the BMG-580, you need to familar with the flash attention algorithm, roofline model, perforamnce analyze tools and hardware architechture. Besides, the optmization process is iterative and time consuming, you need to repeat the workflow multiple times to approache the roofline: collect the performance counters with unitrace tools, find the rootcasue of the hostspot from the insights and then apply the optmization changes.

The agent AI provide a good practice to reduce the mannual effort and make the optmization process more scalable. In this RFC, we provide a top-down methodology to do the performance optmization and will build a skills suite to make this process to be reusable.

Two things make this an RFC rather than a tuning handbook:

1. Knowledge vs Skill are separated. Knowledge is static fact (peak compute, bandwidth, GRF size, DPAS shape) referenced by skills; skills are executable actions (measure or rewrite). Any SKU-specific number lives in knowledge/, never hard-coded in a skill
2. The Skills must be data driven. Data Driven means that the rootcasue should based on the insights from the profier trace and unitrace counter. Performance gain only with hyper parameter tuning w/o insights is not a good practice and should not be part of skills.

---

2. The Top-down Trunk: from Model to ASM

Four abstraction levels, drilled top-down: form a hypothesis on the way down, validate it with measurement on the way up.

Level 0  MODEL   whole-graph timeline → is the model host- or kernel-bound? + Amdahl-rank ops → pick a hot op
   │
Level 1  OP      one given-shape op → host vs kernel split (same test as L0, op scope) → kernel-bound → hand to L2
   │
Level 2  KERNEL  one GPU kernel → roofline ceiling + unitrace counters → compute- vs memory-bound + bottleneck
   │
Level 3  ASM     ISA + ASM↔source mapping → EU-pipe utilization & stall attribution → apply lever

Level 0 and Level 1 run the same host-vs-kernel classifier at two scopes — L0 over the whole graph (to decide which op is worth optimizing), L1 over a single given-shape op (to decide whether that op is host- or kernel-bound; even a tiny elementwise op can be dispatch-bound by itself). The entry point depends on the input: a full model starts at L0; a single op skips straight to L1. Only a kernel-bound op descends into the expensive Level 2/3 analysis (clock locking, single-kernel isolation, multiple replays), so the cheap L0/L1 split protects you from "optimizing the wrong object".

---

3. Trunk Decision Tree

flowchart TD
    IN([input: model or single op]) --> SC{scope?}
    SC -->|whole model| P0[Level 0: whole-graph profiler trace]
    SC -->|single given-shape op| P1

    P0 --> Q0{GPU timeline busy?}
    Q0 -->|many gaps, GPU idle| HB
    Q0 -->|dense, GPU busy| RANK[Amdahl-rank ops] --> P1

    P1[Level 1: one op<br/>host vs kernel split] --> Q1{kernel_time > host_time?}
    Q1 -->|no, host wins| HB[HOST-BOUND branch]
    Q1 -->|yes, kernel wins| A_gate{algorithm gate:<br/>is the algorithm optimal?}

    HB --> HB1{source of host cost?}
    HB1 -->|Python/trace overhead| H_py[python overhead<br/>→ torch.compile / fewer graph breaks]
    HB1 -->|many tiny kernels| H_launch[launch / dispatch bound<br/>→ fusion / SYCL-graph / persistent kernel]
    HB1 -->|sycl/l0 runtime| H_launch[launch / dispatch bound<br/>→ fusion / SYCL-graph ]
    
    A_gate -->|no| A_fix[structural rewrite<br/>flash-attn / online-softmax / split-K]
    A_gate -->|yes| KP[Level 2/3: Kernel Analysis Pipeline<br/>see §3.1]

Three deliberate design choices:

1. The host/kernel split runs at both scopes. L0 over the graph, L1 over a single op; launch-bound and sync-bound live under the host-bound branch, not as top-level classes.
2. The algorithm gate precedes roofline. A structural rewrite (flash-attention class) usually beats any micro-tuning, so ask "is the algorithm even right" before spending on counters.
3. Below the gate, every kernel runs one fixed pipeline (§3.1). The roofline verdict only selects which layered drill-down template applies (step 4 vs 5) — it does not change the first four steps.

---

3.1 Kernel Analysis Pipeline (Level 2/3)

Every kernel-bound op runs the same fixed pipeline. Steps 0–3 are common to all kernels; only the final drill-down (step 4/5) is selected by the roofline verdict.

flowchart TD
    S0["0. build-roofline — algo + HW → layered ceilings DRAM/L2/L1/compute<br/>verdict: compute- vs memory-bound; for memory-bound, theoretical RD/WR per level"]
    S0 --> S1["1. unitrace performance counters (measured)"]
    S1 --> S2["2. ASM extract + ASM↔source mapping"]
    S2 --> S3["3. EU utilization: ALU-pipe util + stall attribution"]
    S3 --> BR{roofline verdict × kernel type}
    BR -->|Non-GEMM kernel| M["4. Memory-hierarchy drill-down<br/>L1 → L2 → DRAM RD/WR, measured vs theoretical, per level"]
    BR -->|GEMM-like kernel| C["5. Tiling-hierarchy drill-down<br/>subgroup → workgroup → mainloop → scheduling"]
    
    M --> LEV[apply lever → re-measure → back to step 0]
    C --> LEV
 

Steps 0–3 (common). (0) Roofline is built statically from the algorithm's FLOPs + bytes and the HW ceilings; for a memory-bound kernel it also emits the theoretical RD/WR at each memory level. (1) unitrace gives the measured counters. (2) ASM + source mapping localizes hot IPs. (3) EU utilization splits into ALU-pipe utilization and stall attribution — this step also catches the latency/occupancy-bound case (MBU and MFU both low, yet no pipe is saturated → TLP occupancy / ILP levers).

Step 4 — memory-hierarchy drill-down (memory-bound). Walk L1 → L2 → DRAM; at each level compare measured bytes (counters) against theoretical bytes (roofline). we need to avoid the memory overfetch, partial write and un-alignment which introduce the unused data load.

Step 5 — tiling-hierarchy drill-down (compute-bound GEMM). Walk the GEMM decomposition top-down, each layer with its own metric and lever:

Layer	Metric	Lever
subgroup tiling	DPAS utilization, GRF pressure / spill	register blocking, map-to-dpas, reduce-grf-spill
workgroup tiling	SLM usage, L1/L2 reuse, occupancy	tile sizing, raise-occupancy
mainloop	prefetch distance, double-buffer, SWSB pipeline stalls	prefetch, double-buffer, overlap-async-copy
scheduling policy	workgroup→Xe-core mapping, load balance, tail effect	launch / scheduling config

---

4. Building-Block Decomposition

4.1 Placing the skills on the architecture matrix

The trunk in §2–3 is a vertical, depth-first traversal (Level 0 model → Level 3 ASM): each level answers "how deep have I drilled?". Baside on the PyTorch software stck, Skills as originized as the following 2-axis matrix:

• Stack-Layer (row) — where in the SW/HW stack a skill operates (Application → Framework → Dependency-Libs → Runtime → Driver → Hardware).
• Abstraction (column) — how composable it is: a Workflow (Tier-1, multi-step, scenario-facing) vs an Atomic (Tier-2, single reusable action).

These two views are not redundant — they are perpendicular. The RFC's Level tells you the order of operations; the matrix's Stack-Layer tells you which component owns the skill, and the Abstraction tells you whether it orchestrates or acts. Concretely, drilling down the trunk traces a diagonal across the matrix: Level 0/1 work lands in the upper rows (Application/Framework — timeline, host/kernel split), the library-bypass (§5) sits in Dependency-Libs, and Level 2/3 work falls into the Hardware row (roofline, counters, ASM, levers). The whole RFC is itself one Tier-1 orchestration workflow (op-kernel-perf-optimization, Application × Workflow).

Mapping every building block from §3/§4 onto the two axes:

Stack-Layer ↓ \ Abstraction →	Workflow (Tier-1, composite)	Atomic (Tier-2, single action)
Application / Scenario	op-kernel-perf-optimization (the trunk, orchestrates all below) · algorithm-review (algorithm gate)
Framework (PyTorch + Eco)	bisect-guilty-commit (regression localization)	measure-op-latency · collect-profiler-trace · triage-timeline · classify-op-bound · decompose-host-overhead · validate-numerics · apply-fusion
Dependency-Libs (Triton/oneDNN/SYCL-TLA)	oneDNN path · Triton path (analyze-triton-kernel)	collect-dnnl-verbose · run-benchdnn · extract-asm-onednn · run-triton-kernel · extract-asm-triton
Runtime (oneAPI/L0/SYCL)	—
Driver / DL-Essential (IGC/UMD/KMD)	—	lock-gpu-clocks
Hardware (Chip/Xe-core/EU/Memory)	eu-utilization-triage · eu-stall-attribution · memory-roofline-breakdown (step 4) · gemm-tiling-analysis (step 5) · eu-ilp-coissue · eu-tlp-occupancy	build-roofline · extract-xpu-kernel-asm · asm-source-mapping · collect-eu-counters · count-grf-spill · memory-workload-analysis · reduce-grf-spill · overlap-async-copy

4.2 Per-phase block detail

The tables below list the same skills again, but ordered by the trunk's execution phases (Level/depth) rather than by stack-layer, and annotate each with its concrete output and referenced knowledge. Read §4.1 for where a skill lives; read §4.2 for when it fires.

Phase 0 — Measurement & Localization (Level 0)

Block	Output	Skill	Knowledge
B0.1 reliable timing	warmup/variance-controlled device & wall time	measure-op-latency (gap)	measurement-methodology
B0.2 lock clocks	fixed GPU frequency (prerequisite for roofline)	lock-gpu-clocks (gap)	hw/by-platform/b580 (clock domain)
B0.3 whole-graph timeline	host+device trace	collect-profiler-trace (have)	—

Phase 1 — Host vs Kernel Split (Level 1)

Block	Output	Skill	Knowledge
B1.1 host/kernel decision	bound category (host/kernel)	classify-op-bound (gap)	—
B1.2 host-stack decomposition	python / dispatch / UR-L0 submit shares	decompose-host-overhead (gap)	sw-stack/framework/dispatch, sw-stack/runtime/sycl-ur
B1.3 launch-bound detection	kernel count / gap distribution	(folded into B0.4 / B1.2)	—

Phase 2 — Algorithm Gate + Roofline (Level 2)

Block	Output	Skill	Knowledge
B2.1 algorithm gate	"is the algorithm optimal" verdict + candidates	algorithm-review (gap, knowledge-driven)	algorithm-patterns (attention/gemm/reduction)
B2.2 roofline build	AI, layered ceilings, ridge point, bound type	build-roofline (gap)	hw/by-platform/b580 (peak compute / per-level bandwidth)

Phase 3 — Kernel pipeline steps 1–5 (Level 2/3)

Maps to §3.1: B3.1–3.5 + B3.7 are the common steps 1–3; B3.6/B3.8 are step 4 (memory-hierarchy); B3.9 is step 5 (tiling-hierarchy).

Block	Output	Skill	Knowledge
B3.1 three-state portrait	stall / single-pipe / co-issue shares → routing	eu-utilization-triage (have)	hw/by-category/xve
B3.2 counter collection	OA / stall-sampling raw counts	collect-eu-counters (have)	—
B3.3 stall attribution	dominant stall class + stage + code ownership	eu-stall-attribution (have)	hw/by-category/swsb
B3.4 ASM extraction	per-kernel ISA	extract-xpu-kernel-asm (have)	—
B3.5 ASM→source mapping	hot IP → file:line	asm-source-mapping (have)	—
B3.6 memory working set	L2/DRAM bytes, hit rate, coalescing	memory-workload-analysis (gap)	hw/by-category/cache-hierarchy
B3.7 occupancy/GRF	active threads, GRF mode, spill count	measure-occupancy / count-grf-spill (gap)	hw/by-category/grf
B3.8 memory-hierarchy drill-down (step 4)	per-level L1→L2→DRAM RD/WR, measured vs theoretical	memory-roofline-breakdown (gap)	hw/by-category/cache-hierarchy
B3.9 GEMM tiling-hierarchy drill-down (step 5)	subgroup/workgroup/mainloop/scheduling metrics	gemm-tiling-analysis (gap)	hw/by-category/{dpas,grf,slm}

Phase 4 — Levers (apply the fix, change code) ← the library's biggest gap

Executable optimization actions. Every lever follows a three-part contract: trigger (which conclusion above fired) → change → verify.

Lever skill	Trigger	Category
apply-fusion	launch-bound / memory-bound (repeated DRAM round-trips)	mem + launch
vectorize-mem-access	memory-bound, uncoalesced/unvectorized access	memory
increase-ilp-coissue	stall: high single-pipe share	latency (eu-ilp-coissue exists)
raise-occupancy	stall: latency not hidden + low occupancy	latency (eu-tlp-occupancy exists)
reduce-grf-spill	spill count > 0 dragging occupancy down	latency
overlap-async-copy	memory latency exposed	latency

Phase 5 — Regression Guard (cross-cutting)

Block	Output	Skill
B5.1 numeric validation	accuracy/correctness regression	validate-numerics (gap)
B5.2 shape sweep	speedup matrix across shapes (avoid single-point overfit)	shape-sweep (gap)
B5.3 causal localization	regression attributed to a commit	bisect-guilty-commit family (have)

---

5. Library-Path Bypass (oneDNN / Triton / Inductor)

Many XPU operators do not land on a hand-written SYCL kernel, but on a oneDNN primitive, Triton (Inductor-generated), or a SYCL-TLA template. At Level 2 the trunk branches to the matching sub-path, reusing existing skills:

flowchart LR
    K[hot kernel] --> W{codegen source?}
    W -->|oneDNN ngen| ON[collect-dnnl-verbose<br/>→ run-benchdnn<br/>→ extract-asm-onednn]
    W -->|Triton/Inductor| TR[analyze-triton-kernel<br/>→ run-triton-kernel<br/>→ extract-asm-triton]
    W -->|hand-written SYCL AOT/JIT| SY[extract-asm-syclkernel-*]
    ON --> CT2[back to Phase 3 counter top-down]
    TR --> CT2
    SY --> CT2

These paths all already have skill implementations; the RFC only hooks them onto the trunk's "source discrimination" node.

---

Alternatives

No response

Additional context

No response