Bazel's use of -isystem gives incorrect search path on macOS
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 72
Description
### Description of the problem:
Bazel's use of -isystem gives incorrect search path on macOS, specifically 10.15.2 (19C57) which is the system I have right now.
The observed behavior is that /usr/local/include is incorrectly searched before the directories local to external dependencies. If a dependency has two distinct versions both installed under /usr/local/include and specified in WORKSPACE as an external dependency, it probably won't compile nicely, or even if it compiles, the resulted binary is probably not what we intend it to be.
### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
1- Install latest version of protobuf with homebrew. This isn't specific to protobuf, but it happened to be one of the dependencies that I encountered.
```
% brew install protobuf
% brew info protobuf
protobuf: stable 3.11.2 (bottled), HEAD
...
```
2- Check out the protobuf examples. Again, this isn't specific to this example. I'm just too lazy to prepare and upload some code snippet somewhere.
```
% git clone https://github.com/protocolbuffers/protobuf.git
% cd protobuf/examples
```
3- Update WORKSPACE to something as follows. I'm just using an older version of protobuf to make the problem evident.
```
workspace(name = "com_google_protobuf_examples")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "com_google_protobuf",
strip_prefix = "protobuf-3.10.1",
url = "https://github.com/protocolbuffers/protobuf/archive/v3.10.1.zip",
sha256 = "678d91d8a939a1ef9cb268e1f20c14cd55e40361dc397bb5881e4e1e532679b1",
)
load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()
```
4- Build the example. The extra flags to bazel are to make this output reproducible as much as I can.
```
% bazel build --jobs 1 --verbose_failures --sandbox_debug :add_person_cpp
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:add_person_cpp (16 packages loaded, 560 targets configured).
INFO: Found 1 target...
ERROR: /private/var/tmp/_bazel_xjia/701b89b752af7feca00b1102287553da/external/com_google_protobuf/BUILD:164:1: C++ compilation of rule '@com_google_protobuf//:protobuf' failed (Exit 1) sandbox-exec failed: error executing command
(cd /private/var/tmp/_bazel_xjia/701b89b752af7feca00b1102287553da/sandbox/darwin-sandbox/5/execroot/com_google_protobuf_examples && \
exec env - \
PATH=/usr/local/opt/bison/bin:/usr/local/lib/ruby/gems/2.6.0/bin:/usr/local/opt/ruby/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin \
PWD=/proc/self/cwd \
TMPDIR=/var/folders/tq/nxwms1d13d57l4ky75sp9fh80000gn/T/ \
/usr/bin/sandbox-exec -f /private/var/tmp/_bazel_xjia/701b89b752af7feca00b1102287553da/sandbox/darwin-sandbox/5/sandbox.sb /var/tmp/_bazel_xjia/install/4f12cbbb3a294bfa314e578758559d54/process-wrapper '--timeout=0' '--kill_delay=15' external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/darwin-fastbuild/bin/external/com_google_protobuf/_objs/protobuf/descriptor.pb.pic.d '-frandom-seed=bazel-out/darwin-fastbuild/bin/external/com_google_protobuf/_objs/protobuf/descriptor.pb.pic.o' -fPIC -iquote external/com_google_protobuf -iquote bazel-out/darwin-fastbuild/bin/external/com_google_protobuf -iquote external/zlib -iquote bazel-out/darwin-fastbuild/bin/external/zlib -isystem external/com_google_protobuf/src -isystem bazel-out/darwin-fastbuild/bin/external/com_google_protobuf/src -isystem external/zlib/zlib/include -isystem bazel-out/darwin-fastbuild/bin/external/zlib/zlib/include -DHAVE_PTHREAD -DHAVE_ZLIB -Woverloaded-virtual -Wno-sign-compare -Wno-unused-function -Wno-write-strings -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc -o bazel-out/darwin-fastbuild/bin/external/com_google_protobuf/_objs/protobuf/descriptor.pb.pic.o)
external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc:1489:52: error: out-of-line definition of 'InternalSerializeWithCachedSizesToArray' does not match any declaration in 'google::protobuf::FileDescriptorSet'
::PROTOBUF_NAMESPACE_ID::uint8* FileDescriptorSet::InternalSerializeWithCachedSizesToArray(
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc:1498:25: error: cannot initialize a parameter of type 'google::protobuf::uint8 *' (aka 'unsigned char *') with an rvalue of type '::google::protobuf::uint8 **' (aka 'unsigned char **')
stream->EnsureSpace(&target);
^~~~~~~
/usr/local/include/google/protobuf/io/coded_stream.h:688:54: note: passing argument to parameter 'ptr' here
PROTOBUF_MUST_USE_RESULT uint8* EnsureSpace(uint8* ptr) {
^
external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc:1500:7: error: no member named 'InternalWriteMessageToArray' in 'google::protobuf::internal::WireFormatLite'; did you mean 'InternalWriteMessage'?
InternalWriteMessageToArray(1, this->_internal_file(i), target, stream);
^~~~~~~~~~~~~~~~~~~~~~~~~~~
InternalWriteMessage
/usr/local/include/google/protobuf/wire_format_lite.h:1710:31: note: 'InternalWriteMessage' declared here
inline uint8* WireFormatLite::InternalWriteMessage(
^
...
```
The exact errors are not significant to this issue. The important part is that the dependency-local files are #include-ing their corresponding header files with angle brackets <...> and the headers are incorrectly resolved to be under /usr/local/include.
### What operating system are you running Bazel on?
macOS Catalina 10.15.2 (19C57)
### What's the output of `bazel info release`?
release 2.0.0
### What's the output of `git remote get-url origin ; git rev-parse master ; git rev-parse HEAD` ?
```
https://github.com/protocolbuffers/protobuf.git
6263268b8c1b78a8a9b65acd6f5dd5c04dd9b0e1
6263268b8c1b78a8a9b65acd6f5dd5c04dd9b0e1
```
### Have you found anything relevant by searching the web?
https://github.com/protocolbuffers/protobuf/issues/5376 is definitely the same issue.
I did searches with "is:issue -isystem /usr/local/include" in the Bazel issues, but the results seemed too noisy for me to correlate any of them to this issue.
### Any other information, logs, or outputs that you want to share?
I debugged this for a while, and my best guess so far is that there are two issues:
1- Bazel puts a space between -isystem and the path, which Clang doesn't seem like to take it that way. It also applies to -iquote.
2- The Clang I have here somehow adds "-I/usr/local/include" automatically. And since -I has higher precedence over -isystem, the search path will be wrong as long as we use -isystem for the external dependency directories.
To verify these two items, I simplified the bazel command line and tried these:
```
xjia@mba examples % cd bazel-examples
xjia@mba bazel-examples % pwd
/Users/xjia/protobuf/examples/bazel-examples
xjia@mba bazel-examples % /Library/Developer/CommandLineTools/usr/bin/cpp -v -iquote external/com_google_protobuf -isystem external/com_google_protobuf/src /dev/null -o /dev/null
Apple clang version 11.0.0 (clang-1100.0.33.16)
Target: x86_64-apple-darwin19.2.0
...
clang: warning: argument unused during compilation: '-traditional' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-iquote external/com_google_protobuf' [-Wunused-command-line-argument]
Apple clang version 11.0.0 (clang-1100.0.33.16)
Target: x86_64-apple-darwin19.2.0
...
clang -cc1 version 11.0.0 (clang-1100.0.33.16) default target x86_64-apple-darwin19.2.0
ignoring nonexistent directory "-isystem"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
/usr/local/include
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/include
/Library/Developer/CommandLineTools/usr/include
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.
error: error reading 'external/com_google_protobuf/src'
1 error generated.
xjia@mba bazel-examples % /Library/Developer/CommandLineTools/usr/bin/cpp -v -iquote external/com_google_protobuf -iquote bazel-out/darwin-fastbuild/bin/external/com_google_protobuf -isystem external/com_google_protobuf/src /dev/null -o /dev/null
...
clang: warning: argument unused during compilation: '-traditional' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-iquote external/com_google_protobuf' [-Wunused-command-line-argument]
...
ignoring nonexistent directory "-iquote"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
/usr/local/include
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/include
/Library/Developer/CommandLineTools/usr/include
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.
error: error reading 'bazel-out/darwin-fastbuild/bin/external/com_google_protobuf'
1 error generated.
xjia@mba bazel-examples % /Library/Developer/CommandLineTools/usr/bin/cpp -v -iquote'external/com_google_protobuf' -isystem'external/com_google_protobuf/src' /dev/null -o /dev/null
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
external/com_google_protobuf
#include <...> search starts here:
/usr/local/include
external/com_google_protobuf/src
/Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/include
/Library/Developer/CommandLineTools/usr/include
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.
# 1 "/dev/null"
# 1 "" 1
# 1 "" 3
# 361 "" 3
# 1 "" 1
# 1 "" 2
# 1 "/dev/null" 2
Apple clang version 11.0.0 (clang-1100.0.33.16)
Target: x86_64-apple-darwin19.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
clang: warning: argument unused during compilation: '-traditional' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-iquote external/com_google_protobuf' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-isystem external/com_google_protobuf/src' [-Wunused-command-line-argument]
xjia@mba bazel-examples % echo $?
0
```
I used /Library/Developer/CommandLineTools/usr/bin/cpp in those commands because I was also suspecting https://github.com/Homebrew/brew/blob/master/Library/Homebrew/shims/super/cc but it looks innocent to me now.
Contributor guide
Research direction
Reproduce the report with the protobuf examples, the shown WORKSPACE entry, and the Bazel build command on macOS. Start by inspecting the emitted Clang arguments and include search order around -iquote, -isystem, and /usr/local/include; done means the dependency-local protobuf headers are selected and the example builds successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, macos
- Domain
- build-system, compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100