EnzymeAD / EnzymeAD/Enzyme-JAX
Shardy all-to-all error
Open
Nobody has claimed this yet.
- Dominant language
- MLIR
- Stars
- 133
- Forks
- 53
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 193
Description
┌ Error: Compilation failed, MLIR module written to /tmp/reactant_sIYaxA/module_000_reactant_first_t..._post_xla_compile.mlir
└ @ Reactant.MLIR.IR /mnt3/wmoses/git/Reactant.jl/src/mlir/IR/Pass.jl:119
┌ Error: Failed to compile
│ exception =
│ INVALID_ARGUMENT: Shardy export for HLO round trip failed;
│
│ Detailed error from MLIR: /home/wmoses/.julia/packages/LLVM/iza6e/src/interop/base.jl:39:0: error: 'sdy.all_to_all' op collective on operand without sharding
│ none:0:0: note: called from
│ none:0:0: note: called from
│ /home/wmoses/.julia/packages/LLVM/iza6e/src/interop/pointer.jl:88:0: note: called from
│ /mnt3/wmoses/git/Reactant.jl/ext/ReactantCUDAExt.jl:354:0: note: called from
│ /mnt3/wmoses/git/Reactant.jl/ext/ReactantCUDAExt.jl:347:0: note: called from
│ /mnt3/wmoses/git/Reactant.jl/ext/ReactantCUDAExt.jl:389:0: note: called from
│ /mnt3/wmoses/git/Reactant.jl/ext/ReactantCUDAExt.jl:401:0: note: called from
│ /home/wmoses/.julia/packages/Oceananigans/CGCJ5/src/BoundaryConditions/fill_halo_regions_periodic.jl:81:0: note: called from
│ /home/wmoses/.julia/packages/KernelAbstractions/X5fk1/src/extras/loopinfo.jl:31:0: note: called from
│ /home/wmoses/.julia/packages/Oceananigans/CGCJ5/src/BoundaryConditions/fill_halo_regions_periodic.jl:78:0: note: called from
│ /home/wmoses/.julia/packages/LLVM/iza6e/src/interop/base.jl:39:0: note: see current operation: %2115 = "sdy.all_to_all"(%2106) <{out_sharding = #sdy.sharding<@mesh, [{"x"}, {"y"}, {}]>,
params = #sdy<all_to_all_param_list[{"x"}: 2->0]>}> : (tensor<4x1520x3054xf32>) -> tensor<4x1520x3054xf32>
│
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the generated MLIR module referenced in /tmp and the stack-trace locations in ext/ReactantCUDAExt.jl, especially lines 347-401. Trace the call into Oceananigans/src/BoundaryConditions/fill_halo_regions_periodic.jl and determine how to reproduce the sdy.all_to_all error. The issue does not define the expected fix, so completion would require confirming the intended behavior and adding a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100