OOM in XLA when compiling functions

While compiling the first time step of the sharded baroclinic model at https://github.com/PRONTOLab/GB-25/blob/f2763e7527a90bf5712851585d0bae7e36b37564/sharding/sharded_baroclinic_instability_simulation_run.jl#L66 with a large enough grid (e.g. 3056×1520×512) we're experiencing large compile time and/or OOM. Using `earlyoom` to send a SIGTERM when memory runs low we get the following stacktrace:
```
[85140] signal 15: Terminated
in expression starting at /capstor/scratch/cscs/mgiordan/santis/repo/GB-25/sharding/runs/2025-04-14T18-26-21.178_wyOe/sharded_baroclinic_instability_simulation_run.jl:66
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:148
start_thread at /lib64/libpthread.so.0 (unknown line)
thread_start at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:148
start_thread at /lib64/libpthread.so.0 (unknown line)
thread_start at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:148
start_thread at /lib64/libpthread.so.0 (unknown line)
thread_start at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:148
start_thread at /lib64/libpthread.so.0 (unknown line)
thread_start at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:148
start_thread at /lib64/libpthread.so.0 (unknown line)
thread_start at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:148
start_thread at /lib64/libpthread.so.0 (unknown line)
thread_start at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
jl_parallel_gc_threadfun at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:148
start_thread at /lib64/libpthread.so.0 (unknown line)
thread_start at /lib64/libc.so.6 (unknown line)
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
pthread_cond_wait at /lib64/libpthread.so.0 (unknown line)
uv_cond_wait at /workspace/srcdir/libuv/src/unix/thread.c:822
ijl_task_get_next at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/scheduler.c:584
poptask at ./task.jl:1012
wait at ./task.jl:1021
task_done_hook at ./task.jl:694
jfptr_task_done_hook_66648 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
jl_finish_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:319
start_task at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/task.c:1213
unknown function (ip: (nil))
_ZN3xla14HloInstruction5Users6UserIdEPS0_ at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla14HloInstruction5Users10RemoveUserEPS0_ at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla14HloInstruction28ReplaceUseWithDifferentShapeEPS0_S1_ at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla14HloInstruction32ReplaceAllUsesWithDifferentShapeEN4absl12lts_202308024SpanIKPS0_EES4_ at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla14HloInstruction15ReplaceUsesWithEN4absl12lts_202308024SpanIKPS0_EES4_ at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla12_GLOBAL__N_126LayoutNormalizationVisitor13DefaultActionEPNS_14HloInstructionE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla28DfsHloVisitorWithDefaultBaseIPNS_14HloInstructionEE13HandleBitcastES2_ at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla14HloInstruction5VisitIPS0_EEN4absl12lts_202308026StatusEPNS_17DfsHloVisitorBaseIT_EE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xlaL12PostOrderDFSINS_17DfsHloVisitorBaseIPNS_14HloInstructionEEEEEN4absl12lts_202308026StatusES3_PT_St8optionalINS6_11FunctionRefIFbSt4pairIiPKS2_ESF_EEEEbb.isra.0 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla14HloInstruction6AcceptIPS0_EEN4absl12lts_202308026StatusEPNS_17DfsHloVisitorBaseIT_EEbbb at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZNK3xla14HloComputation6AcceptIPNS_14HloInstructionEEEN4absl12lts_202308026StatusEPNS_17DfsHloVisitorBaseIT_EE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla19LayoutNormalization3RunEPNS_9HloModuleERKN4absl12lts_2023080213flat_hash_setISt17basic_string_viewIcSt11char_traitsIcEENS4_18container_internal10StringHashENSA_8StringEqESaIS9_EEE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla15HloPassPipeline17RunPassesInternalINS_9HloModuleEEEN4absl12lts_202308028StatusOrIbEEPT_RKNS_12DebugOptionsERKNS4_13flat_hash_setISt17basic_string_viewIcSt11char_traitsIcEENS4_18container_internal10StringHashENSH_8StringEqESaISG_EEE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla15HloPassPipeline3RunEPNS_9HloModuleERKN4absl12lts_2023080213flat_hash_setISt17basic_string_viewIcSt11char_traitsIcEENS4_18container_internal10StringHashENSA_8StringEqESaIS9_EEE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla16HloPassInterface3RunEPNS_9HloModuleE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla3gpu11GpuCompiler17OptimizeHloModuleEPNS_9HloModuleEPN15stream_executor14StreamExecutorERKNS_8Compiler14CompileOptionsERKNS7_12TargetConfigE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla3gpu11GpuCompiler12RunHloPassesESt10unique_ptrINS_9HloModuleESt14default_deleteIS3_EEPN15stream_executor14StreamExecutorERKNS_8Compiler14CompileOptionsE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla12LLVMCompiler7CompileESt10unique_ptrINS_14HloModuleGroupESt14default_deleteIS2_EESt6vectorIS6_IPN15stream_executor14StreamExecutorESaIS9_EESaISB_EERKNS_8Compiler14CompileOptionsE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla7Service16BuildExecutablesERKSt6vectorIPKNS_14HloModuleProtoESaIS4_EES1_ISt10unique_ptrINS_15HloModuleConfigESt14default_deleteISA_EESaISD_EEPNS_7BackendES1_IS1_IPN15stream_executor14StreamExecutorESaISK_EESaISM_EERKNS_8Compiler14CompileOptionsEb at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla12LocalService18CompileExecutablesERKNS_14XlaComputationEN4absl12lts_202308024SpanIKPKNS_5ShapeEEERKNS_22ExecutableBuildOptionsE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla11LocalClient7CompileERKNS_14XlaComputationEN4absl12lts_202308024SpanIKPKNS_5ShapeEEERKNS_22ExecutableBuildOptionsE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla24PjRtStreamExecutorClient15CompileInternalERKNS_14XlaComputationERKSt6vectorIPKNS_5ShapeESaIS7_EESt8functionIFN4absl12lts_202308028StatusOrISt4pairIS4_IS5_SaIS5_EES5_EEERKNS_9HloModuleEEENS_14CompileOptionsE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla24PjRtStreamExecutorClient7CompileEN4mlir8ModuleOpENS_14CompileOptionsE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla24PjRtStreamExecutorClient14CompileAndLoadEN4mlir8ModuleOpENS_14CompileOptionsE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla4ifrt20PjRtLoadedExecutable6CreateEPNS0_20PjRtCompatibleClientEN4mlir8ModuleOpENS_14CompileOptionsESt6vectorIN3tsl11RCReferenceINS0_18LoadedHostCallbackEEESaISB_EE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
_ZN3xla4ifrt12PjRtCompiler7CompileESt10unique_ptrINS0_7ProgramESt14default_deleteIS3_EES2_INS0_14CompileOptionsES4_IS7_EE at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
ifrt_compile at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/dev/Reactant/deps/ReactantExtra/bazel-bin/libReactantExtra.so (unknown line)
#33 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/packages/Reactant/ABoJI/src/xla/IFRT/LoadedExecutable.jl:89
try_compile_dump_mlir at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/packages/Reactant/ABoJI/src/mlir/IR/Pass.jl:131
try_compile_dump_mlir at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/packages/Reactant/ABoJI/src/mlir/IR/Pass.jl:126 [inlined]
#compile#32 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/packages/Reactant/ABoJI/src/xla/IFRT/LoadedExecutable.jl:88 [inlined]
compile at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/packages/Reactant/ABoJI/src/xla/IFRT/LoadedExecutable.jl:74
unknown function (ip: 0x40150f270267)
#compile_xla#54 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/packages/Reactant/ABoJI/src/Compiler.jl:2801
compile_xla at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/packages/Reactant/ABoJI/src/Compiler.jl:2753 [inlined]
#compile#55 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/packages/Reactant/ABoJI/src/Compiler.jl:2824
compile at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/packages/Reactant/ABoJI/src/Compiler.jl:2823
unknown function (ip: 0x40151d7901f7)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
do_call at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/interpreter.c:126
eval_value at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/interpreter.c:223
eval_stmt_value at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/interpreter.c:174 [inlined]
eval_body at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/interpreter.c:684
jl_interpret_toplevel_thunk at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/interpreter.c:824
jl_toplevel_eval_flex at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/toplevel.c:943
jl_toplevel_eval_flex at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/toplevel.c:886
ijl_toplevel_eval_in at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/toplevel.c:994
eval at ./boot.jl:430 [inlined]
include_string at ./loading.jl:2734
_include at ./loading.jl:2794
include at ./Base.jl:557
jfptr_include_46855 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
exec_options at ./client.jl:323
_start at ./client.jl:531
jfptr__start_73550 at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/lib/julia/sys.so (unknown line)
jl_apply at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/julia.h:2157 [inlined]
true_main at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/jlapi.c:900
jl_repl_entrypoint at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/src/jlapi.c:1059
main at /cache/build/default-aws-aarch64-ci-1-2/julialang/julia-release-1-dot-11/cli/loader_exe.c:58
__libc_start_main at /lib64/libc.so.6 (unknown line)
_start at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/bin/julia (unknown line)
_start at /capstor/scratch/cscs/mgiordan/santis/juliaup/depot/juliaup/julia-1.11.4+0.aarch64.linux.gnu/bin/julia (unknown line)
unknown function (ip: (nil))
Allocations: 300377779 (Pool: 300369311; Big: 8468); GC: 134
process 85140 did not exit
kill failed: Timer expired
mem avail: 67218 of 452187 MiB (14.87%), swap free: 0 of 0 MiB ( 0.00%)
low memory! at or below SIGTERM limits: mem 15.00%, swap 100.00%
```

Dumps from Reactant and XLA:
* [mlir_dumps.tar.gz](https://github.com/user-attachments/files/19741941/mlir_dumps.tar.gz)
* [xla_dump.tar.gz](https://github.com/user-attachments/files/19741940/xla_dump.tar.gz)