Bazel does not handle symlinks in system include paths for CUDA
- Dominant language
- Java
- Stars
- 25.8k
- Forks
- 4.6k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 75
Description
### Description of the problem / feature request:
Very similar to https://github.com/bazelbuild/bazel/issues/7315:
If the compilers system includes are installed in a Path which is referenced by a symlink then the dependency check will fail.
Example:
- Compiler includes in real path `/software/haswell/GCCcore/8.2.0/include/c++/8.2.0/x86_64-pc-linux-gnu /8.2.0/include`
- Symlink exists so user sees `/sw/installed/GCCcore/8.2.0/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include/`
- Toolchain file (e.g. created by TensorFlow for CUDA) will contain the 2nd one (resolved path, `/sw/installed`)
- Check will use unresolved path and check only the prefix in https://github.com/bazelbuild/bazel/blob/c42f93f2285e0d8f1ea70ccf3168200db34235f5/src/main/java/com/google/devtools/build/lib/rules/cpp/HeaderDiscovery.java#L141
- Error will be something like:
```
ERROR: /tmp/easybuild-tmp/eb-5QGVSJ/tmpMZolLj-bazel-build/external/fft2d/BUILD.bazel:27:1: undeclared inclusion(s) in rule '@fft2d//:fft2d':
this rule is missing dependency declarations for the following files included by 'external/fft2d/fft2d/fftsg2d.c':
'/sw/installed/GCCcore/8.2.0/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include/stddef.h'
'/sw/installed/GCCcore/8.2.0/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include/stdarg.h'
Target //tensorflow/tools/pip_package:build_pip_package failed to build
```
### Feature requests: what underlying problem are you trying to solve with this feature?
Building TensorFlow on a HPC cluster. See related issue https://github.com/tensorflow/tensorflow/issues/33975
### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
See above: Setup the compiler with a symlink path and try to build tensorflow 2 with CUDA
### What operating system are you running Bazel on?
RHEL 7.4
### What's the output of `bazel info release`?
0.26.1 (@non-git)
### If `bazel info release` returns "development version" or "(@non-git)", tell us how you built Bazel.
With EasyBuild recipe
### Have you found anything relevant by searching the web?
- TensorFlow issue https://github.com/tensorflow/tensorflow/issues/33975
- Bazel issue https://github.com/bazelbuild/bazel/issues/7315
- SO questions like https://stackoverflow.com/questions/35256110/tensorflow-build-fails-with-missing-dependency-error/35274720
- Accepted answer does not apply (anymore)
- Using `-MMD` to exclude system headers from dependencies might work (2nd answer)
Possible solutions:
1. patch TensorFlow to include non-resolved paths: https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/t/TensorFlow/TensorFlow-1.14.0_fix-cuda-build.patch
2. Unify logic of https://github.com/tensorflow/tensorflow/blob/ce69fe54695361d88c3b73d18e0d647baeb50c49/third_party/gpus/cuda_configure.bzl#L280 with https://github.com/bazelbuild/bazel/blob/8b0bfaf6716327ca6b60692704e3d57766f7ab1f/tools/cpp/unix_cc_configure.bzl#L125 as indicated at https://github.com/tensorflow/tensorflow/blob/ce69fe54695361d88c3b73d18e0d647baeb50c49/third_party/gpus/cuda_configure.bzl#L221 and handle it in bazel (resolved and non-resolved paths added)
3. Check resolved dependencies against each other in https://github.com/bazelbuild/bazel/blob/c42f93f2285e0d8f1ea70ccf3168200db34235f5/src/main/java/com/google/devtools/build/lib/rules/cpp/HeaderDiscovery.java
Contributor guide
Research direction
Start by reading HeaderDiscovery.java at the referenced dependency-check logic, then compare it with unix_cc_configure.bzl and TensorFlow's cuda_configure.bzl. Reproduce a CUDA/TensorFlow build with compiler include paths reached through a symlink on RHEL 7.4. Done means system headers reached through resolved and unresolved paths no longer produce undeclared inclusion errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100