bazelbuild / bazelbuild/bazel

collect2: fatal error: cannot find 'ld' with build --incompatible_strict_action_env

Open
#21,334 16 comments 2 reactions 0 assignees View on GitHub
P3 team-Rules-CPP type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 20h
Merged PRs (30d)
72

Description

### Description of the bug:

Seems to be related to #20834

When I enable `build --incompatible_strict_action_env` in .bazelrc building our project fails with the following error:

```
➜ kotlin git:(agluszak/refactor-spaghetti) ✗ bazel build //... --sandbox_debug
INFO: Analyzed 150 targets (0 packages loaded, 7 targets configured).
DEBUG: Sandbox debug output for CppLink @@protobuf~21.7//:protoc:
1707844200.104456969: src/main/tools/linux-sandbox.cc:156: calling pipe(2)...
1707844200.104511302: src/main/tools/linux-sandbox.cc:165: Netns is 0
1707844200.104513667: src/main/tools/linux-sandbox.cc:176: calling clone(2)...
1707844200.105022932: src/main/tools/linux-sandbox.cc:185: linux-sandbox-pid1 has PID 80651
1707844200.105051827: src/main/tools/linux-sandbox-pid1.cc:700: Pid1Main started
1707844200.105319123: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/execroot -> /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-execroot
1707844200.105360231: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/execroot -> /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-working-directory
1707844200.105373727: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/external/bazel_tools -> /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-source-roots/0
1707844200.105393635: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/external/bazel_tools~cc_configure_extension~local_config_cc -> /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-source-roots/1
1707844200.105407010: src/main/tools/linux-sandbox-pid1.cc:302: bind mount: /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp -> /tmp
1707844200.105425235: src/main/tools/linux-sandbox-pid1.cc:311: writable: /dev/shm
1707844200.105436987: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp
1707844200.105454781: src/main/tools/linux-sandbox-pid1.cc:311: writable: /tmp/bazel-execroot/_main
1707844200.105465631: src/main/tools/linux-sandbox-pid1.cc:327: working dir: /tmp/bazel-working-directory/_main
1707844200.105580890: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /
1707844200.105589215: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev
1707844200.105595107: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/pts
1707844200.105601248: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1707844200.105606588: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/hugepages
1707844200.105612490: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /dev/mqueue
1707844200.105619072: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run
1707844200.105625053: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/lock
1707844200.105630794: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/credentials/systemd-sysusers.service
1707844200.105636886: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/qemu
1707844200.105642567: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000
1707844200.105649129: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/gvfs
1707844200.105692892: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/user/1000/doc
1707844200.105699324: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /run/docker/netns/f38912d93c2c
1707844200.105703352: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /run/docker/netns/f38912d93c2c, nullptr, 4129, nullptr) failure (Permission denied) ignored
1707844200.105712730: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys
1707844200.105718380: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/firmware/efi/efivars
1707844200.105729321: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/security
1707844200.105750030: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/cgroup
1707844200.105757975: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/pstore
1707844200.105764367: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/bpf
1707844200.105770820: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/debug
1707844200.105777412: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/tracing
1707844200.105784055: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/fs/fuse/connections
1707844200.105791569: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /sys/kernel/config
1707844200.105798192: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc
1707844200.105803432: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1707844200.105813982: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /proc/sys/fs/binfmt_misc, nullptr, 2101281, nullptr) failure (Operation not permitted) ignored
1707844200.105854238: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /proc/sys/fs/binfmt_misc
1707844200.105861712: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /recovery
1707844200.105866732: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /boot/efi
1707844200.105891278: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /var/lib/docker/overlay2/f8d24e86254fc80e1069e49e0e17b4ac571ff497cb87bf0590dbd5d86ec9add3/merged
1707844200.105896468: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /var/lib/docker/overlay2/f8d24e86254fc80e1069e49e0e17b4ac571ff497cb87bf0590dbd5d86ec9add3/merged, nullptr, 2101281, nullptr) failure (Permission denied) ignored
1707844200.105899704: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/.mount_jetbrasqy1gx
1707844200.105915123: src/main/tools/linux-sandbox-pid1.cc:427: remount(nullptr, /tmp/.mount_jetbrasqy1gx, nullptr, 2101287, nullptr) failure (No such file or directory) ignored
1707844200.105919051: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-execroot
1707844200.105925743: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-working-directory
1707844200.105950019: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-source-roots/0
1707844200.105956882: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-source-roots/1
1707844200.105962754: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1707844200.105967643: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-execroot
1707844200.105972612: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-working-directory
1707844200.105977221: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/0
1707844200.105982391: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/1
1707844200.105987691: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /dev/shm
1707844200.105992610: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp
1707844200.105997099: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-execroot
1707844200.106009552: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-working-directory
1707844200.106014261: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/0
1707844200.106019261: src/main/tools/linux-sandbox-pid1.cc:405: remount ro: /tmp/bazel-source-roots/1
1707844200.106024090: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp/bazel-execroot/_main
1707844200.106029079: src/main/tools/linux-sandbox-pid1.cc:405: remount rw: /tmp/bazel-working-directory/_main
1707844200.106194844: src/main/tools/linux-sandbox.cc:202: done manipulating pipes
1707844200.106516263: src/main/tools/linux-sandbox-pid1.cc:496: calling fork...
1707844200.106983849: src/main/tools/linux-sandbox-pid1.cc:533: child started with PID 2
1707844200.131119984: src/main/tools/linux-sandbox-pid1.cc:550: wait returned pid=2, status=0x100
1707844200.131138990: src/main/tools/linux-sandbox-pid1.cc:568: child exited normally with code 1
1707844200.131745710: src/main/tools/linux-sandbox.cc:243: child exited normally with code 1

Run this command to start an interactive shell in an identical sandboxed environment:
(exec env - \
PATH=/bin:/usr/bin:/usr/local/bin \
PWD=/proc/self/cwd \
TMPDIR=/tmp \
ZERO_AR_DATE=1 \
/home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/install/14fb027596f626f2526df4873ea20b8b/linux-sandbox -W /tmp/bazel-working-directory/_main -t 15 -w /dev/shm -w /tmp -w /tmp/bazel-execroot/_main -M /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/execroot -m /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-execroot -M /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/execroot -m /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-working-directory -M /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/external/bazel_tools -m /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-source-roots/0 -M /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/external/bazel_tools~cc_configure_extension~local_config_cc -m /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-source-roots/1 -M /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp -m /tmp -S /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/stats.out -D /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/debug.out -- /bin/sh -i)
ERROR: /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/external/protobuf~21.7/BUILD.bazel:457:10: Linking external/protobuf~21.7/protoc [for tool] failed: (Exit 1): linux-sandbox failed: error executing CppLink command
(cd /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/execroot/_main && \
exec env - \
PATH=/bin:/usr/bin:/usr/local/bin \
PWD=/proc/self/cwd \
TMPDIR=/tmp \
ZERO_AR_DATE=1 \
/home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/install/14fb027596f626f2526df4873ea20b8b/linux-sandbox -W /tmp/bazel-working-directory/_main -t 15 -w /dev/shm -w /tmp -w /tmp/bazel-execroot/_main -M /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/execroot -m /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-execroot -M /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/execroot -m /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-working-directory -M /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/external/bazel_tools -m /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-source-roots/0 -M /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/external/bazel_tools~cc_configure_extension~local_config_cc -m /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp/bazel-source-roots/1 -M /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/_hermetic_tmp -m /tmp -S /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/stats.out -D /home/andrzej.gluszak/.cache/bazel/_bazel_andrzej.gluszak/1c60d9a8d822a2b05451e359eb75781d/sandbox/linux-sandbox/319/debug.out -- /usr/bin/gcc @bazel-out/k8-opt-exec-ST-13d3ddad9198/bin/external/protobuf~21.7/protoc-2.params)
collect2: fatal error: cannot find 'ld'
compilation terminated.
INFO: Elapsed time: 0.190s, Critical Path: 0.04s
INFO: 2 processes: 2 internal.
ERROR: Build did NOT complete successfully
```

Code is available here: https://github.com/JetBrains/bazel-bsp/commit/8a876006fafacf53a5bcd9e574af5724b42e2189

### Which category does this issue belong to?

C++ Rules, Core, Local Execution

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

_No response_

### Which operating system are you running Bazel on?

Ubuntu

### What is the output of `bazel info release`?

release 7.0.2

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

_No response_

### What's the output of `git remote get-url origin; git rev-parse HEAD` ?

_No response_

### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

_No response_

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

_No response_

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with `bazel build //... --incompatible_strict_action_env --sandbox_debug`, then inspect `.bazelrc` and the external protobuf target reported at `BUILD.bazel:457:10`. Compare the sandbox environment and linker lookup with and without the strict action environment. Done means the build completes without the `cannot find 'ld'` error.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.