Tuolumne GPU Deployment

When deploying to Tuolumne (Python 3.11.5 ROCm 6.0.0) this error results when running any GPU program. Environments built using attached source file build_rocm.sh.txt.

Going to retry with Python/3.10.8.

Env

• Python 3.11.8
• Numba 0.60.0
• Numba-hip 0.1.0
• ROCm 6.0.0
• rocm-llvm-python 6.0.0.470.19
• hip-python 6.0.0.470.16
• harmonize (main branch)
• MC/DC Release (0.11.1)

Error and Traceback

(venv-tuo) [morgan83@tuolumne1002:kobayashi3-TD]$ flux run -N 1 -n 1 -g 1 --queue=pdebug python input.py --mode=numba --target=gpu --gpu_arena_size=100000000 --gpu_strat=event
flux-job: faHx86k6rw9 started                                                                                                                                                                                                       00:00:05
 [ VERBOSE ]  ROCM_PLATFORM detected. Proceeding under ROCM.
 [ VERBOSE ]  Found hipcc located at '/opt/rocm-6.0.0/bin/hipcc'
 [ VERBOSE ]  ROCM_PATH set to /opt/rocm-6.0.0
 [ VERBOSE ]  Compiling function 'initialize' for 'mcdc_source_event'
...
 [ VERBOSE ]  Creating shared object file
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: /opt/rocm-6.0.0/llvm/bin/ld.lld --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -shared -o __harmonize_cache__/harmonize.so /lib/../lib64/crti.o /opt/rocm-6.0.0/lib/llvm/lib/clang/17.0.0/lib/linux/clang_rt.crtbegin-x86_64.o -L/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12 -L/opt/rh/gcc-toolset-12/root/usr/lib/gcc/x86_64-redhat-linux/12/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/lib -L/usr/lib --disable-new-dtags /var/tmp/morgan83/harmonize-d20c5b.o /var/tmp/morgan83/harmonize-1b417a.o -L/opt/rocm-6.0.0/lib/llvm/bin/../../../lib -rpath /opt/rocm-6.0.0/lib/llvm/bin/../../../lib -lamdhip64 -lstdc++ -lm /opt/rocm-6.0.0/lib/llvm/lib/clang/17.0.0/lib/linux/libclang_rt.builtins-x86_64.a -lgcc_s -lc /opt/rocm-6.0.0/lib/llvm/lib/clang/17.0.0/lib/linux/libclang_rt.builtins-x86_64.a -lgcc_s /opt/rocm-6.0.0/lib/llvm/lib/clang/17.0.0/lib/linux/clang_rt.crtend-x86_64.o /lib/../lib64/crtn.o
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  ld.lld          0x00000000008854e8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 40
1  ld.lld          0x0000000000882f0c
2  libpthread.so.0 0x0000155554d40d10
3  ld.lld          0x00000000009d4edc void lld::elf::ELFFileBase::init<llvm::object::ELFType<(llvm::support::endianness)1, true>>(lld::elf::InputFile::Kind) + 364
4  ld.lld          0x0000000000992a66 lld::elf::LinkerDriver::addFile(llvm::StringRef, bool) + 646
5  ld.lld          0x0000000000993b7f lld::elf::LinkerDriver::addLibrary(llvm::StringRef) + 79
6  ld.lld          0x0000000000993f2b lld::elf::LinkerDriver::createFiles(llvm::opt::InputArgList&) + 827
7  ld.lld          0x00000000009aa854 lld::elf::LinkerDriver::linkerMain(llvm::ArrayRef<char const*>) + 1908
8  ld.lld          0x00000000009ac33a lld::elf::link(llvm::ArrayRef<char const*>, llvm::raw_ostream&, llvm::raw_ostream&, bool, bool) + 3706
9  ld.lld          0x000000000080c82c
10 ld.lld          0x000000000080cc56 lld_main(int, char**, llvm::ToolContext const&) + 390
11 ld.lld          0x000000000077aef1 main + 33
12 libc.so.6       0x00001555534ec7e5 __libc_start_main + 229
13 ld.lld          0x000000000080a65e _start + 46
clang: error: unable to execute command: Bus error (core dumped)
clang: error: linker command failed due to signal (use -v to see invocation)
Traceback (most recent call last):
  File "/usr/WS1/morgan83/tuo_dep/MCDC/examples/fixed_source/kobayashi3-TD/input.py", line 70, in <module>
    mcdc.run()
  File "/usr/WS1/morgan83/tuo_dep/MCDC/mcdc/main.py", line 67, in run
    data_arr, mcdc_arr = prepare()
                         ^^^^^^^^^
  File "/usr/WS1/morgan83/tuo_dep/MCDC/mcdc/main.py", line 776, in prepare
    build_gpu_progs(input_deck, config.args)
  File "/usr/WS1/morgan83/tuo_dep/MCDC/mcdc/loop.py", line 856, in build_gpu_progs
    adapt.harm.RuntimeSpec.bind_specs()
  File "/usr/WS1/morgan83/tuo_dep/harmonize/harmonize/python/runtime.py", line 1274, in bind_specs
    subprocess.run(cmd.split(),shell=False,check=True)
  File "/collab/usr/gapps/python/toss_4_x86_64_ib/anaconda3-2023.09/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/opt/rocm-6.0.0/bin/hipcc', '-fPIC', '-shared', '-fgpu-rdc', '--hip-link', '__harmonize_cache__/harmonize.bc', '-o', '__harmonize_cache__/harmonize.so', '-g']' returned non-zero exit status 1.
flux-job: task(s) exited with exit code 1

Start with the attached build_rocm.sh.txt and reproduce the failure using the listed Tuolumne, Python, ROCm, Numba, and harmonize versions; also compare the planned Python 3.10.8 retry. Trace the GPU build path through mcdc/main.py, mcdc/loop.py, and harmonize/python/runtime.py, focusing on the hipcc linker invocation. Done should identify the cause of the ld.lld bus error and establish a verified deployment outcome.