Build failure on NetBSD/sparc64 when using the embedded version of LLVM

I tried to build the rust compiler, version 1.73.0, "natively" on NetBSD/sparc64 10.0_BETA, but that failed with an "illegal instruction" fault.
It's entirely conceivable that this issue is or was present in earlier versions as well.

I expected to see this happen: to see the build succeed.

Instead, this happened: the build failed, with the compiler bombing out with SIGILL ("illegal instruction") in the stage2 phase of the build, with:

     Running `/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/release/build/proc-macro-test-0016d4c2c13207b7/build-script-build`
error: failed to run custom build command for `proc-macro-test v0.0.0 (/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/src/tools/rust-analyzer/crates/proc-macro-test)`

Caused by:
  process didn't exit successfully: `/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/release/build/proc-macro-test-0016d4c2c13207b7/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=imp
  cargo:rerun-if-env-changed=PROC_MACRO_TEST_TOOLCHAIN
  Creating /usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/sparc64-unknown-netbsd/release/build/proc-macro-test-d774d9ce9a192410/out/proc-macro-test-imp-staging
  Creating /usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/sparc64-unknown-netbsd/release/build/proc-macro-test-d774d9ce9a192410/out/proc-macro-test-imp-staging/src
  Copying /usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/src/tools/rust-analyzer/crates/proc-macro-test/imp/Cargo.toml to /usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/sparc64-unknown-netbsd/release/build/proc-macro-test-d774d9ce9a192410/out/proc-macro-test-imp-staging/Cargo.toml
  Copying /usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/src/tools/rust-analyzer/crates/proc-macro-test/imp/src/lib.rs to /usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/sparc64-unknown-netbsd/release/build/proc-macro-test-d774d9ce9a192410/out/proc-macro-test-imp-staging/src/lib.rs
  Running cd "/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/sparc64-unknown-netbsd/release/build/proc-macro-test-d774d9ce9a192410/out/proc-macro-test-imp-staging" && "/usr/pkgsrc/wip/rust/work/rust-bootstrap/bin/cargo" "build" "-p" "proc-macro-test-impl" "--message-format" "json" "--target-dir" "/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/sparc64-unknown-netbsd/release/build/proc-macro-test-d774d9ce9a192410/out/target"
  proc-macro-test-impl failed to build
  ============ stdout ============

  Did not run successfully: signal: 4 (SIGILL) (core dumped)
  LD_LIBRARY_PATH="/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2/lib:/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/sparc64-unknown-netbsd/release/build/proc-macro-test-d774d9ce9a192410/out/target/debug/deps:/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2/lib:/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/release/deps:/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/release:/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2/lib/rustlib/sparc64-unknown-netbsd/lib:/usr/pkgsrc/wip/rust/work/rust-bootstrap/lib" "/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2/bin/rustc" "--crate-name" "proc_macro_test_impl" "--edition=2021" "src/lib.rs" "--error-format=json" "--json=diagnostic-rendered-ansi,artifacts,future-incompat" "--crate-type" "proc-macro" "--emit=dep-info,link" "-C" "prefer-dynamic" "-C" "embed-bitcode=no" "-C" "debuginfo=2" "-C" "metadata=0b44ceb97b41db90" "-C" "extra-filename=-0b44ceb97b41db90" "--out-dir" "/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/sparc64-unknown-netbsd/release/build/proc-macro-test-d774d9ce9a192410/out/target/debug/deps" "-L" "dependency=/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/sparc64-unknown-netbsd/release/build/proc-macro-test-d774d9ce9a192410/out/target/debug/deps" "--extern" "proc_macro" "-Csymbol-mangling-version=v0" "-Zunstable-options" "--check-cfg=values(bootstrap)" "-Zmacro-backtrace" "-Clink-args=-Wl,-z,origin" "-Clink-args=-Wl,-rpath,/usr/pkg/lib" "-Zunstable-options" "-Csplit-debuginfo=off" "-Wrust_2018_idioms" "-Wunused_lifetimes" "-Wsemicolon_in_expressions_from_macros" "-Dwarnings" "-Zunstable-options" "--check-cfg=values(bootstrap)" "-Zallow-features=binary-dep-depinfo,proc_macro_span,proc_macro_span_shrink,proc_macro_diagnostic,proc_macro_internals,proc_macro_diagnostic,proc_macro_span,proc_macro_span_shrink"
  -------------
  {"reason":"build-finished","success":false}

  ============ stderr ============
     Compiling proc-macro-test-impl v0.0.0 (/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/sparc64-unknown-netbsd/release/build/proc-macro-test-d774d9ce9a192410/out/proc-macro-test-imp-staging)
  rustc exited with signal: 4 (SIGILL) (core dumped)
  error: could not compile `proc-macro-test-impl` (lib)

  Caused by:
    process didn't exit successfully: `/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/bootstrap/debug/rustc --crate-name proc_macro_test_impl --edition=2021 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type proc-macro --emit=dep-info,link -C prefer-dynamic -C embed-bitcode=no -C debuginfo=2 -C metadata=0b44ceb97b41db90 -C extra-filename=-0b44ceb97b41db90 --out-dir /usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/sparc64-unknown-netbsd/release/build/proc-macro-test-d774d9ce9a192410/out/target/debug/deps -L dependency=/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools/sparc64-unknown-netbsd/release/build/proc-macro-test-d774d9ce9a192410/out/target/debug/deps --extern proc_macro -Csymbol-mangling-version=v0 -Zunstable-options '--check-cfg=values(bootstrap)' -Zmacro-backtrace -Clink-args=-Wl,-z,origin -Clink-args=-Wl,-rpath,/usr/pkg/lib -Zunstable-options -Csplit-debuginfo=off` (exit status: 254)


  --- stderr
  thread 'main' panicked at crates/proc-macro-test/build.rs:83:9:
  proc-macro-test-impl failed to build
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

command did not execute successfully: cd "/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src" && AR_sparc64_unknown_netbsd="ar" CARGO_INCREMENTAL="0" CARGO_PROFILE_RELEASE_DEBUG="0" CARGO_PROFILE_RELEASE_DEBUG_ASSERTIONS="false" CARGO_PROFILE_RELEASE_OVERFLOW_CHECKS="false" CARGO_TARGET_DIR="/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2-tools" CC_sparc64_unknown_netbsd="gcc" CFG_COMPILER_HOST_TRIPLE="sparc64-unknown-netbsd" CFG_RELEASE="1.73.0" CFG_RELEASE_CHANNEL="stable" CFG_RELEASE_NUM="1.73.0" CFG_VERSION="1.73.0 (cc66ad468 2023-10-03) (built from a source tarball)" CFG_VER_DATE="2023-10-03" CFG_VER_HASH="cc66ad468955717ab92600c770da8c1601a4ff33" CFLAGS_sparc64_unknown_netbsd="-O2 -ffunction-sections -fdata-sections -fPIC -O2 -fPIC -D_FORTIFY_SOURCE=2 -I/usr/pkg/include -I/usr/include/krb5 -I/usr/include -I/usr/pkg/include/python3.10" CXXFLAGS_sparc64_unknown_netbsd="-O2 -ffunction-sections -fdata-sections -fPIC -O2 -fPIC -D_FORTIFY_SOURCE=2 -I/usr/pkg/include -I/usr/include/krb5 -I/usr/include -I/usr/pkg/include/python3.10" CXX_sparc64_unknown_netbsd="c++" DOC_RUST_LANG_ORG_CHANNEL="https://doc.rust-lang.org/1.73.0" LIBC_CHECK_CFG="1" LZMA_API_STATIC="1" RANLIB_sparc64_unknown_netbsd="ar s" REAL_LIBRARY_PATH="/usr/pkgsrc/wip/rust/work/rust-bootstrap/lib" REAL_LIBRARY_PATH_VAR="LD_LIBRARY_PATH" RUSTBUILD_NATIVE_DIR="/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/native" RUSTC="/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/bootstrap/debug/rustc" RUSTC_ALLOW_FEATURES="binary-dep-depinfo,proc_macro_span,proc_macro_span_shrink,proc_macro_diagnostic,proc_macro_internals,proc_macro_diagnostic,proc_macro_span,proc_macro_span_shrink" RUSTC_BOOTSTRAP="1" RUSTC_BREAK_ON_ICE="1" RUSTC_ERROR_METADATA_DST="/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/tmp/extended-error-metadata" RUSTC_INSTALL_BINDIR="bin" RUSTC_LIBDIR="/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2/lib" RUSTC_LINT_FLAGS="-Wrust_2018_idioms -Wunused_lifetimes -Wsemicolon_in_expressions_from_macros -Dwarnings" RUSTC_REAL="/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2/bin/rustc" RUSTC_SNAPSHOT="/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2/bin/rustc" RUSTC_SNAPSHOT_LIBDIR="/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2/lib" RUSTC_STAGE="2" RUSTC_SYSROOT="/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2" RUSTC_TLS_MODEL_INITIAL_EXEC="1" RUSTC_VERBOSE="1" RUSTDOC="/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/bootstrap/debug/rustdoc" RUSTDOCFLAGS="-Csymbol-mangling-version=v0 -Zunstable-options --check-cfg=values(bootstrap) -Dwarnings -Wrustdoc::invalid_codeblock_attributes --crate-version 1.73.0\t(cc66ad468\t2023-10-03)\t(built\tfrom\ta\tsource\ttarball)" RUSTDOC_REAL="/path/to/nowhere/rustdoc/not/required" RUSTFLAGS="-Csymbol-mangling-version=v0 -Zunstable-options --check-cfg=values(bootstrap) -Zmacro-backtrace -Clink-args=-Wl,-z,origin -Clink-args=-Wl,-rpath,/usr/pkg/lib -Zunstable-options -Csplit-debuginfo=off" RUST_TEST_THREADS="1" SYSROOT="/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/build/sparc64-unknown-netbsd/stage2" __CARGO_DEFAULT_LIB_METADATA="stable" "/usr/pkgsrc/wip/rust/work/rust-bootstrap/bin/cargo" "build" "--target" "sparc64-unknown-netbsd" "--release" "-Zcheck-cfg=names,values,output,features" "-Zbinary-dep-depinfo" "-j" "1" "-v" "--frozen" "--manifest-path" "/usr/pkgsrc/wip/rust/work/rustc-1.73.0-src/src/tools/rust-analyzer/Cargo.toml" "--features" "rust-analyzer/in-rust-tree"
expected success, got: exit status: 101

Meta

Hmm, the rustc version used to bootstrap this is (obviously) the previous rust version, 1.72.1, which is cross-built from another platform (NetBSD/amd64):

s1: {17} ls -l work
total 40
drwxr-xr-x  12 he  wsrc  1024 Sep 24 17:48 rust-1.72.0-sparc64-unknown-netbsd
drwxr-xr-x   6 he  wsrc   512 Oct 24 03:19 rust-bootstrap
drwxr-xr-x   3 he  wsrc   512 Sep 24 17:32 rust-std-1.72.0-sparc64-unknown-netbsd
drwxr-xr-x   9 he  wsrc   512 Oct 24 03:24 rustc-1.73.0-src
drwxr-xr-x   2 he  wsrc   512 Oct 24 03:22 scripts
s1: {18} 

To test a theory that this particular problem stems from the embedded LLVM, I re-tried the build by using an external LLVM, version 15.0.7, and with that configuration, the build succeeds.

It's conceivable that I will re-test this with LLVM version 16.something, but it will at a minimum take multiple days to complete that test.

Sorry, I did not collect the backtrace for this problem, though it's not a given that it would have fared much better than producing the log above. My sparc64 host isn't exactly a speed demon, so the lead time for a test is in the order of days.

I have also tried to look at the core dump with gdb on this system, but regrettably didn't get very far with that -- I believe there are ... "regressions in our gdb port" which makes this difficult.

For now I'm asking if this is something other users of the sparc64 CPU platform are seeing, and whether anyone has any hints about this particular problem.

Start with the stage2 build failure in src/tools/rust-analyzer/crates/proc-macro-test/build.rs and the reported SIGILL while compiling proc-macro-test-impl on NetBSD/sparc64. Reproduce the native build with embedded LLVM, then compare it with the successful configuration using external LLVM 15.0.7. Done means identifying and fixing the embedded-LLVM cause, with the stage2 build succeeding on this target.