bazelbuild / bazelbuild/rules_rust

Non-deterministic behavior with sandboxing disabled

Open
#1,244 0 comments 0 reactions 0 assignees View on GitHub
bug needs-triage
Dominant language
Starlark
Stars
843
Forks
651
Avg merge
2d 18h
Merged PRs (30d)
15

Description

For some projects there is a [performance benefit](https://github.com/bazelbuild/bazel/issues/8230) gained by disabling sandboxing. With sandboxing disabled when I integrated rules_rust + [cxx](https://github.com/dtolnay/cxx) into a project I found that there is some behavior that breaks with sandboxing disabled in interesting ways. You end up with lots of linker errors, either duplicate symbols, missing symbols, empty files, or linker crashes (at least with ld.gold). Here's an example from macOS:

```
ERROR: /Users/ksmiley/dev/cxx/third-party/BUILD:141:19: Compiling Rust bin syn@build_ (1 files) [for host] failed: (Exit 1): process_wrapper failed: error executing command bazel-out/host/bin/external/rules_rust/util/process_wrapper/process_wrapper --subst 'pwd=${pwd}' -- external/rust_darwin_aarch64/bin/rustc third-party/vendor/syn-1.0.86/build.rs '--crate-name=build' ... (remaining 30 arguments skipped)
error: linking with `external/local_config_cc/cc_wrapper.sh` failed: exit status: 134
|
= note: "external/local_config_cc/cc_wrapper.sh" "-arch" "arm64" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.0.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.1.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.10.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.11.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.12.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.13.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.14.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.15.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.2.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.3.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.4.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.5.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.6.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.7.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.8.rcgu.o" "bazel-out/host/bin/third-party/build.build.7698221b-cgu.9.rcgu.o" "bazel-out/host/bin/third-party/build.4503yh7u17l6jf67.rcgu.o" "-L" "external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib" "-L" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/libstd-81655915c211065a.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/libpanic_unwind-6401d7836ab37fcf.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/libobject-bee3b45bdece0195.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/libmemchr-01c74ed3833459fe.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/libaddr2line-48661015cf0226dd.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/libgimli-f6cb12e379c9f859.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/libstd_detect-01c2377d8875d7d3.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/librustc_demangle-4ecebe9133c1e15e.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/libhashbrown-68b6457fbd59457b.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_alloc-9d4889633473e617.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/libunwind-7d202cbf8f30fd4c.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/libcfg_if-412e60c7c12b3b8f.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/liblibc-45ccb8237a0071b3.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/liballoc-24424da3181053ad.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_core-5789d203d1806d35.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/libcore-22574ec029e9d229.rlib" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-56bb43a121401f7f.rlib" "-lSystem" "-lresolv" "-lc" "-lm" "-liconv" "-L" "/private/var/tmp/_bazel_ksmiley/b40bedb0fdd7aea108909e499cde2302/external/rust_darwin_aarch64/lib/rustlib/aarch64-apple-darwin/lib" "-o" "bazel-out/host/bin/third-party/syn@build_" "-Wl,-dead_strip" "-nodefaultlibs" "-lc++" "-fobjc-link-runtime" "-headerpad_max_install_names" "-no-canonical-prefixes" "-target" "arm64-apple-macosx" "-mmacosx-version-min=12.1" "-lc++" "-target" "arm64-apple-macosx"
= note: Undefined symbols for architecture arm64:
"alloc::collections::btree::navigate::_$LT$impl$u20$alloc..collections..btree..node..Handle$LT$alloc..collections..btree..node..NodeRef$LT$alloc..collections..btree..node..marker..Dying$C$K$C$V$C$alloc..collections..btree..node..marker..Leaf$GT$$C$alloc..collections..btree..node..marker..Edge$GT$$GT$::deallocating_next_unchecked::h1b3d7bba7c60370c", referenced from:
_$LT$alloc..collections..btree..map..BTreeMap$LT$K$C$V$GT$$u20$as$u20$core..ops..drop..Drop$GT$::drop::hc34cd945a465fa96 in build.build.7698221b-cgu.3.rcgu.o
"_$LT$core..iter..sources..empty..Empty$LT$T$GT$$u20$as$u20$core..iter..traits..iterator..Iterator$GT$::next::h850925f87a589837", referenced from:
l_anon.fad58de7366495db4650cfefac2fcd61.52 in build.build.7698221b-cgu.0.rcgu.o
"_$LT$core..iter..sources..empty..Empty$LT$T$GT$$u20$as$u20$core..iter..traits..iterator..Iterator$GT$::size_hint::h38c1a019137d1cd7", referenced from:
l_anon.fad58de7366495db4650cfefac2fcd61.52 in build.build.7698221b-cgu.0.rcgu.o
"_anon.925bc04f7ce27dffcbeb5532418f12b7.0.llvm.18185687318075157643", referenced from:
_main in build.build.7698221b-cgu.0.rcgu.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error in child process '/usr/bin/xcrun'. 1
external/local_config_cc/cc_wrapper.sh: line 69: 78905 Abort trap: 6 "$(/usr/bin/dirname "$0")"/wrapped_clang "$@"
```

This appears to be because some object files are left around in bazel's output directory, and when they're not cleaned up by sandboxing, they pollute the next build (I'm not sure what tool is responsible for auto-discovering these files, I'm assuming it's `rustc`). Ideally rules_rust would cleanup these files, or execute actions in a temporary directory so they don't cause conflicts in subsequent builds.

This can be reproduced by cloning [cxx](https://github.com/dtolnay/cxx) and running `bazel build demo:all --spawn_strategy=local` (if you forget to change the spawn strategy and get a successful build, run `bazel clean` and then build again)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.