[AMDGPU] Shared-half coalescing serializes independent BF16 loads on gfx1100
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
On gfx1100, two independent BF16 loads can become serialized when each value is packed into the upper half of an FP32 value. Coalescing a COPY of their shared low half also merges the two high-half load results, introducing a register anti-dependence before machine scheduling.
This is a missed optimization, not a numerical correctness failure. Reproduced on upstream main at [`12b2a6767a17`](https://github.com/llvm/llvm-project/commit/12b2a6767a17e6450a3e844359afa30440bdd9ba), built as Release with assertions, targeting `gfx1100`.
### Reproducer
Save as `repro.ll` and run:
```sh
llc -O3 -mcpu=gfx1100 -verify-machineinstrs repro.ll -o -
```
```llvm
target triple = "amdgpu11.00-amd-amdhsa"
define amdgpu_kernel void @silu(ptr addrspace(1) %x, ptr addrspace(1) %gate, ptr addrspace(1) %out) {
%i = call i32 @llvm.amdgcn.workitem.id.x()
%xp = getelementptr bfloat, ptr addrspace(1) %x, i32 %i
%gp = getelementptr bfloat, ptr addrspace(1) %gate, i32 %i
%op = getelementptr bfloat, ptr addrspace(1) %out, i32 %i
%xb = load bfloat, ptr addrspace(1) %xp, align 2
%xpack = insertelement <2 x bfloat> , bfloat %xb, i32 1
%xf = bitcast <2 x bfloat> %xpack to float
%scaled = fmul float %xf, 0xBFF7154760000000
%ex = call float @llvm.amdgcn.exp2.f32(float %scaled)
%denom = fadd float %ex, 1.0
%sig = fdiv float 1.0, %denom
%silu = fmul float %xf, %sig
%gb = load bfloat, ptr addrspace(1) %gp, align 2
%gpack = insertelement <2 x bfloat> , bfloat %gb, i32 1
%gf = bitcast <2 x bfloat> %gpack to float
%result = fmul float %silu, %gf
%bf = fptrunc float %result to bfloat
store bfloat %bf, ptr addrspace(1) %op, align 2
ret void
}
declare i32 @llvm.amdgcn.workitem.id.x()
declare float @llvm.amdgcn.exp2.f32(float)
```
The resulting instruction order is:
```asm
global_load_d16_hi_b16 v1, v0, s[0:1] ; x
s_waitcnt vmcnt(0)
; ...
v_exp_f32_e32 v2, v2
; ... division and multiply ...
global_load_d16_hi_b16 v1, v0, s[2:3] ; gate, reusing v1
s_waitcnt vmcnt(0)
```
Both loads could instead be issued before the first wait / exponent calculation. The second load only contributes to the final multiplication.
### Where the dependency appears
Inspecting the reproducer with `-stop-after=register-coalescer` shows both VMEM instructions defining `%51.hi16`. Before coalescing, the two loaded values are distinct and are packed with a shared low-half zero. Eliminating the COPY of that shared half merges their containing 32-bit virtual registers. The resulting WAR dependency prevents the scheduler from hoisting the second load across the remaining uses of the first value.
A targeted experiment that keeps the shared-half COPY, while still allowing the individual load-to-half copies to coalesce, leaves distinct VMEM results and lets the existing scheduler issue the loads early. The `global_load_d16_hi_b16` selection is retained.
### Proposed change
Keep the shared-half COPY when the other halves have distinct single VMEM definitions in the same basic block. This lets the existing scheduler issue both loads early, at the cost of a retained COPY and one additional VGPR in this reproducer (8 to 9). Profitability on pressure-sensitive workloads remains a review consideration.
Assisted-by: OpenAI Codex
Contributor guide
Research direction
Save the supplied LLVM IR as repro.ll and run the listed llc command for the gfx1100 target. Inspect the register-coalescer output and the machine scheduler around the two global_load_d16_hi_b16 instructions, focusing on the shared low-half COPY. Done means the independent loads can be issued before the first wait while machine verification still passes and the selected load form is retained.
Written by the indexing model from the issue text.
Assessment
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100