JuliaGPU / JuliaGPU/AMDGPU.jl

@ROCStaticLocalArray's default zeroinit costs ~11x: every lane memsets the whole LDS array at kernel entry

Open
#1,055 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Julia
Stars
344
Forks
79
Avg merge
2d 23h
Merged PRs (30d)
27

Description

@ROCStaticLocalArray (and @ROCDynamicLocalArray) default to zeroinit=true, and the zero-initialization is implemented in a way that costs an order of magnitude on any kernel that uses it.

Measurement (MI300A, ROCm 7.2.4, Julia 1.12.6): a 256-thread LDS tree reduction over 2²⁶ Float32, identical code except the macro argument:

time effective bandwidth
@ROCStaticLocalArray(Float32, 256) (default) 2,122.8 µs 126 GB/s
@ROCStaticLocalArray(Float32, 256, false) 187.8 µs 1,430 GB/s

Why it is this expensive. zeroinit_lds! (src/compiler/zeroinit_lds.jl) inserts a memset of the whole LDS global at kernel entry followed by s_barrier. Every work-item executes that memset, so for a 1 KiB array the ISA is 64 × ds_write_b128 per wave, all 64 lanes writing the same address each time (ds_write_b128 v1, v[2:5] offset:N with v1 = 0), which serializes on bank conflicts — and that's repeated by every wave in the workgroup, on every launch. The prologue dwarfs the reduction itself.

Related surfaces

  • KernelAbstractions.@localmem (KA.SharedMemoryalloc_special without zeroinit) and src/kernels/mapreduce.jl (@ROCDynamicLocalArray(T, items, false)) already avoid it, so library code is fine; only the user-facing macros pay by default.
  • @ROCDynamicLocalArray's zeroinit is a per-work-item serial loop over the whole array (for idx in 1:prod(dims); DA[idx] = zero(T)), same pathology.
  • CUDA.jl's CuStaticSharedArray does not zero shared memory; neither does HIP's __shared__.

Suggested fix: default zeroinit=false (matching CUDA.jl/HIP semantics; document that LDS contents are undefined), and if zeroing is requested, distribute it — each work-item clears its strided slice (for i in tid:groupsize:len) before the barrier — instead of every lane clearing everything.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/compiler/zeroinit_lds.jl and trace how @ROCStaticLocalArray and @ROCDynamicLocalArray request zero-initialization. Compare the existing false paths, then validate the reported reduction benchmark. Done means zero-initialization is opt-in, requested clearing is distributed across work-items, and the undefined-LDS behavior is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.