bazelbuild / bazelbuild/bazel

Bazel does not handle symlinks in system include paths for CUDA

Open
#10,167 7 comments 0 reactions 0 assignees View on GitHub
not stale P4 team-Rules-CPP type: bug
Dominant language
Java
Stars
25.8k
Forks
4.6k
Avg merge
2d 18h
Merged PRs (30d)
75

Description

### Description of the problem / feature request:

Very similar to https://github.com/bazelbuild/bazel/issues/7315:
If the compilers system includes are installed in a Path which is referenced by a symlink then the dependency check will fail.

Example:
- Compiler includes in real path `/software/haswell/GCCcore/8.2.0/include/c++/8.2.0/x86_64-pc-linux-gnu /8.2.0/include`
- Symlink exists so user sees `/sw/installed/GCCcore/8.2.0/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include/`
- Toolchain file (e.g. created by TensorFlow for CUDA) will contain the 2nd one (resolved path, `/sw/installed`)
- Check will use unresolved path and check only the prefix in https://github.com/bazelbuild/bazel/blob/c42f93f2285e0d8f1ea70ccf3168200db34235f5/src/main/java/com/google/devtools/build/lib/rules/cpp/HeaderDiscovery.java#L141
- Error will be something like:
```
ERROR: /tmp/easybuild-tmp/eb-5QGVSJ/tmpMZolLj-bazel-build/external/fft2d/BUILD.bazel:27:1: undeclared inclusion(s) in rule '@fft2d//:fft2d':
this rule is missing dependency declarations for the following files included by 'external/fft2d/fft2d/fftsg2d.c':
'/sw/installed/GCCcore/8.2.0/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include/stddef.h'
'/sw/installed/GCCcore/8.2.0/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include/stdarg.h'
Target //tensorflow/tools/pip_package:build_pip_package failed to build
```

### Feature requests: what underlying problem are you trying to solve with this feature?

Building TensorFlow on a HPC cluster. See related issue https://github.com/tensorflow/tensorflow/issues/33975

### Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

See above: Setup the compiler with a symlink path and try to build tensorflow 2 with CUDA

### What operating system are you running Bazel on?

RHEL 7.4

### What's the output of `bazel info release`?

0.26.1 (@non-git)

### If `bazel info release` returns "development version" or "(@non-git)", tell us how you built Bazel.

With EasyBuild recipe

### Have you found anything relevant by searching the web?

- TensorFlow issue https://github.com/tensorflow/tensorflow/issues/33975
- Bazel issue https://github.com/bazelbuild/bazel/issues/7315
- SO questions like https://stackoverflow.com/questions/35256110/tensorflow-build-fails-with-missing-dependency-error/35274720
- Accepted answer does not apply (anymore)
- Using `-MMD` to exclude system headers from dependencies might work (2nd answer)

Possible solutions:
1. patch TensorFlow to include non-resolved paths: https://github.com/easybuilders/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/t/TensorFlow/TensorFlow-1.14.0_fix-cuda-build.patch
2. Unify logic of https://github.com/tensorflow/tensorflow/blob/ce69fe54695361d88c3b73d18e0d647baeb50c49/third_party/gpus/cuda_configure.bzl#L280 with https://github.com/bazelbuild/bazel/blob/8b0bfaf6716327ca6b60692704e3d57766f7ab1f/tools/cpp/unix_cc_configure.bzl#L125 as indicated at https://github.com/tensorflow/tensorflow/blob/ce69fe54695361d88c3b73d18e0d647baeb50c49/third_party/gpus/cuda_configure.bzl#L221 and handle it in bazel (resolved and non-resolved paths added)
3. Check resolved dependencies against each other in https://github.com/bazelbuild/bazel/blob/c42f93f2285e0d8f1ea70ccf3168200db34235f5/src/main/java/com/google/devtools/build/lib/rules/cpp/HeaderDiscovery.java

Contributor guide

Open the contributing guide

Research direction

Start by reading HeaderDiscovery.java at the referenced dependency-check logic, then compare it with unix_cc_configure.bzl and TensorFlow's cuda_configure.bzl. Reproduce a CUDA/TensorFlow build with compiler include paths reached through a symlink on RHEL 7.4. Done means system headers reached through resolved and unresolved paths no longer produce undeclared inclusion errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
build-system
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.