@ROCStaticLocalArray's default zeroinit costs ~11x: every lane memsets the whole LDS array at kernel entry
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 344
- Forks
- 79
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 27
Description
@ROCStaticLocalArray (and @ROCDynamicLocalArray) default to zeroinit=true, and the zero-initialization is implemented in a way that costs an order of magnitude on any kernel that uses it.
Measurement (MI300A, ROCm 7.2.4, Julia 1.12.6): a 256-thread LDS tree reduction over 2²⁶ Float32, identical code except the macro argument:
| time | effective bandwidth | |
|---|---|---|
@ROCStaticLocalArray(Float32, 256) (default) |
2,122.8 µs | 126 GB/s |
@ROCStaticLocalArray(Float32, 256, false) |
187.8 µs | 1,430 GB/s |
Why it is this expensive. zeroinit_lds! (src/compiler/zeroinit_lds.jl) inserts a memset of the whole LDS global at kernel entry followed by s_barrier. Every work-item executes that memset, so for a 1 KiB array the ISA is 64 × ds_write_b128 per wave, all 64 lanes writing the same address each time (ds_write_b128 v1, v[2:5] offset:N with v1 = 0), which serializes on bank conflicts — and that's repeated by every wave in the workgroup, on every launch. The prologue dwarfs the reduction itself.
Related surfaces
KernelAbstractions.@localmem(KA.SharedMemory→alloc_specialwithout zeroinit) andsrc/kernels/mapreduce.jl(@ROCDynamicLocalArray(T, items, false)) already avoid it, so library code is fine; only the user-facing macros pay by default.@ROCDynamicLocalArray's zeroinit is a per-work-item serial loop over the whole array (for idx in 1:prod(dims); DA[idx] = zero(T)), same pathology.- CUDA.jl's
CuStaticSharedArraydoes not zero shared memory; neither does HIP's__shared__.
Suggested fix: default zeroinit=false (matching CUDA.jl/HIP semantics; document that LDS contents are undefined), and if zeroing is requested, distribute it — each work-item clears its strided slice (for i in tid:groupsize:len) before the barrier — instead of every lane clearing everything.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/compiler/zeroinit_lds.jl and trace how @ROCStaticLocalArray and @ROCDynamicLocalArray request zero-initialization. Compare the existing false paths, then validate the reported reduction benchmark. Done means zero-initialization is opt-in, requested clearing is distributed across work-items, and the undefined-LDS behavior is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100