bazel-contrib / bazel-contrib/rules_foreign_cc

How to use the cmake rules and local nvcc compiler to build CUDA project?

Open
#1,322 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Starlark
Stars
737
Forks
270
PR merge metrics
No merged PRs in 30d

Description

I have a cuda project like https://github.com/NVIDIA-Merlin/HierarchicalKV/ , I can build it with it's cmake build guide.
Now I want to use this repo in my Bazel project, so I choose rules_foreign_cc's cmake rule to build it.
Here is my scripts,
WORKSPACE file
```
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
name = "rules_foreign_cc",
sha256 = "a2e6fb56e649c1ee79703e99aa0c9d13c6cc53c8d7a0cbb8797ab2888bbc99a3",
strip_prefix = "rules_foreign_cc-0.12.0",
url = "https://github.com/bazelbuild/rules_foreign_cc/releases/download/0.12.0/rules_foreign_cc-0.12.0.tar.gz",
)

load("@rules_foreign_cc//foreign_cc:repositories.bzl", "rules_foreign_cc_dependencies")

# This sets up some common toolchains for building targets. For more details, please see
# https://bazelbuild.github.io/rules_foreign_cc/0.12.0/flatten.html#rules_foreign_cc_dependencies
rules_foreign_cc_dependencies()

load("@bazel_features//:deps.bzl", "bazel_features_deps")

bazel_features_deps()

http_archive(
name = "HierarchicalKV",
build_file = "hkv.BUILD",
sha256 = "a73d7bea159173db2038f7c5215a7d1fbd5362adfb232fabde206dc64a1e817c",
strip_prefix = "HierarchicalKV-0.1.0-beta.12",
url = "https://github.com/NVIDIA-Merlin/HierarchicalKV/archive/refs/tags/v0.1.0-beta.12.tar.gz",
)
```

hkv.BUILD
```
load("@rules_foreign_cc//foreign_cc:defs.bzl", "cmake")

package(default_visibility = ["//visibility:public"])

licenses(["notice"]) # BSD 3-Clause

exports_files(["LICENSE"])

filegroup(
name = "all_srcs",
srcs = glob(["**"]),
)

cmake(
name = "hkv",
build_args = [
"--verbose",
"-j `nproc`",
],
cache_entries = {
"CMAKE_BUILD_TYPE": "Release",
"BUILD_TESTS": "OFF",
"BUILD_BENCHMARKS": "OFF",
"sm": "80",
},
lib_source = ":all_srcs",
)
```

BUILD file:
```
load("@bazel_skylib//rules:build_test.bzl", "build_test")

build_test(
name = "build_test",
targets = [
"@HierarchicalKV//:hkv",
],
)
```

When I build the target with `bazel build build_test`, I got this error:
```
INFO: Found 1 target...
ERROR: /workspaces/cache/bazel/external/HierarchicalKV/BUILD.bazel:14:6: Foreign Cc - CMake: Building hkv failed: (Exit 1): process-wrapper failed: error executing command
(cd /workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray && \
exec env - \
CLANG_CUDA_COMPILER_PATH=/usr/lib/llvm-17/bin/clang \
HTTPS_PROXY=http://127.0.0.1:7890 \
HTTP_PROXY=http://127.0.0.1:7890 \
LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/:/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 \
PATH=/root/.cache/bazelisk/downloads/sha256/a40ac69263440761199fcb8da47ad4e3f328cbe79ffbf4ecc14e5ba252857307/bin:/bin:/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/l
ocal/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
PYTHON_BIN_PATH=/bin/python3 \
PYTHON_LIB_PATH=/usr/lib/python3/dist-packages \
TF2_BEHAVIOR=1 \
TMPDIR=/tmp \
/root/.cache/bazel/_bazel_root/install/20da5ab742b8d3d499c34fdafcd3c8b8/process-wrapper '--timeout=0' '--kill_delay=15' '--stats=/workspaces/cache/bazel/sandbox/processwrapper-
sandbox/19/stats.out' /bin/bash -c bazel-out/k8-opt/bin/external/HierarchicalKV/hkv_foreign_cc/wrapper_build_script.sh)
rules_foreign_cc: Build failed!
rules_foreign_cc: Keeping temp build directory and dependencies directory for debug.
rules_foreign_cc: Please note that the directories inside a sandbox are still cleaned unless you specify --sandbox_debug Bazel command line flag.
rules_foreign_cc: Printing build logs:
_____ BEGIN BUILD LOGS _____

Bazel external C/C++ Rules. Building library hkv

Environment:______________
BUILD_SCRIPT=bazel-out/k8-opt/bin/external/HierarchicalKV/hkv_foreign_cc/build_script.sh
EXT_BUILD_ROOT=/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray
BUILD_LOG=bazel-out/k8-opt/bin/external/HierarchicalKV/hkv_foreign_cc/CMake.log
PWD=/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray
BUILD_WRAPPER_SCRIPT=bazel-out/k8-opt/bin/external/HierarchicalKV/hkv_foreign_cc/wrapper_build_script.sh
PYTHON_BIN_PATH=/bin/python3
TMPDIR=/tmp
EXT_BUILD_DEPS=/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt/bin/external/HierarchicalKV/hkv.ext_build_deps
BUILD_TMPDIR=/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt/bin/external/HierarchicalKV/hkv.build_tmpdir
SHLVL=2
HTTPS_PROXY=http://127.0.0.1:7890
HTTP_PROXY=http://127.0.0.1:7890
LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/:/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
CLANG_CUDA_COMPILER_PATH=/usr/lib/llvm-17/bin/clang
INSTALLDIR=/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt/bin/external/HierarchicalKV/hkv
PYTHON_LIB_PATH=/usr/lib/python3/dist-packages
TF2_BEHAVIOR=1
PATH=/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray:/root/.cache/bazelisk/downloads/sha256/a40ac69263440761199fcb8da47ad4e3f328cbe79ffbf4ecc14e5ba2528
57307/bin:/bin:/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/bin/env
__________________________
+ /workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/external/cmake-3.22.1-linux-x86_64/bin/cmake -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=OFF -D[21/1930]
CHMARKS=OFF -Dsm=80 -DCMAKE_TOOLCHAIN_FILE=crosstool_bazel.cmake -DCMAKE_INSTALL_PREFIX=/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-op
t/bin/external/HierarchicalKV/hkv -DCMAKE_PREFIX_PATH=/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt/bin/external/HierarchicalKV/hkv.
ext_build_deps -DCMAKE_RANLIB= -DCMAKE_MAKE_PROGRAM=/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt-exec-50AE0418/bin/external/rules_f
oreign_cc/toolchains/make/bin/make -G 'Unix Makefiles' /workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/external/HierarchicalKV
-- The CXX compiler identification is Clang 17.0.6
-- The CUDA compiler identification is NVIDIA 12.2.140
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/lib/llvm-17/bin/clang - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - failed
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc
-- Check for working CUDA compiler: /usr/local/cuda/bin/nvcc - broken
CMake Error at /workspaces/cache/bazel/external/cmake-3.22.1-linux-x86_64/share/cmake-3.22/Modules/CMakeTestCUDACompiler.cmake:56 (message):
The CUDA compiler

"/usr/local/cuda/bin/nvcc"

is not able to compile a simple test program.

It fails with the following output:

Change Dir: /workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt/bin/external/HierarchicalKV/hkv.build_tmpdir/CMakeFiles/CMakeTmp

Run Build Command(s):/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt-exec-50AE0418/bin/external/rules_foreign_cc/toolchains/make/b
in/make -f Makefile cmTC_1738c/fast && /workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt-exec-50AE0418/bin/external/rules_foreign_cc/too
lchains/make/bin/make -f CMakeFiles/cmTC_1738c.dir/build.make CMakeFiles/cmTC_1738c.dir/build
make[1]: Entering directory '/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt/bin/external/HierarchicalKV/hkv.build_tmpdir/CMakeFil
es/CMakeTmp'
Building CUDA object CMakeFiles/cmTC_1738c.dir/main.cu.o
/usr/local/cuda/bin/nvcc -forward-unknown-to-host-compiler -MD -MT CMakeFiles/cmTC_1738c.dir/main.cu.o -MF CMakeFiles/cmTC_1738c.dir/main.cu.o.d -x cu -c /workspaces/cache
/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt/bin/external/HierarchicalKV/hkv.build_tmpdir/CMakeFiles/CMakeTmp/main.cu -o CMakeFiles/cmTC_1738c.dir/m
ain.cu.o
Linking CUDA executable cmTC_1738c
/workspaces/cache/bazel/external/cmake-3.22.1-linux-x86_64/bin/cmake -E cmake_link_script CMakeFiles/cmTC_1738c.dir/link.txt --verbose=1
/bin/g++ -Wl,-no-as-needed -lstdc++ -no-canonical-prefixes -pie -Wl,-z,relro,-z,now -Wl,--gc-sections -Wl,--build-id=md5 -Wl,--hash-style=gnu --cuda-path=/workspaces/cache/ba
zel/sandbox/processwrapper-sandbox/19/execroot/deepray/external/cuda_nvcc -fuse-ld=lld -lm -fuse-ld=lld -lm CMakeFiles/cmTC_1738c.dir/main.cu.o -o cmTC_1738c -lcudadevrt -lcuda
rt_static -lrt -lpthread -ldl -L"/usr/local/cuda/targets/x86_64-linux/lib/stubs" -L"/usr/local/cuda/targets/x86_64-linux/lib"
g++: error: unrecognized command-line option '--cuda-path=/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/external/cuda_nvcc'
make[1]: *** [CMakeFiles/cmTC_1738c.dir/build.make:100: cmTC_1738c] Error 1
make[1]: Leaving directory '/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt/bin/external/HierarchicalKV/hkv.build_tmpdir/CMakeFile
s/CMakeTmp'
make: *** [Makefile:127: cmTC_1738c/fast] Error 2

CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
CMakeLists.txt:17 (project)

-- Configuring incomplete, errors occurred!
See also "/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt/bin/external/HierarchicalKV/hkv.build_tmpdir/CMakeFiles/CMakeOutput.log".
See also "/workspaces/cache/bazel/sandbox/processwrapper-sandbox/19/execroot/deepray/bazel-out/k8-opt/bin/external/HierarchicalKV/hkv.build_tmpdir/CMakeFiles/CMakeError.log".
_____ END BUILD LOGS _____
rules_foreign_cc: Build wrapper script location: bazel-out/k8-opt/bin/external/HierarchicalKV/hkv_foreign_cc/wrapper_build_script.sh
rules_foreign_cc: Build script location: bazel-out/k8-opt/bin/external/HierarchicalKV/hkv_foreign_cc/build_script.sh
rules_foreign_cc: Build log location: bazel-out/k8-opt/bin/external/HierarchicalKV/hkv_foreign_cc/CMake.log

Target //:build_test failed to build
INFO: Elapsed time: 7.980s, Critical Path: 5.65s
INFO: 4 processes: 4 internal.
FAILED: Build did NOT complete successfully
```
So I want to know how to set my local nvcc compiler for the cmake rule

Other infomations:
OS version: Ubuntu 22.04
clang version: Ubuntu clang version 17.0.6
nvcc version: release 12.2, V12.2.140
bazel version: 6.5.0

Contributor guide

Open the contributing guide

Research direction

Start with hkv.BUILD and the WORKSPACE configuration, then inspect CMakeLists.txt at line 17 and the generated CMake.log and CMakeError.log. Reproduce with the shown `bazel build build_test` command and determine how the CMake rule selects the local nvcc compiler; done means the CUDA compiler check and the external HierarchicalKV build succeed.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake
Domain
build-system, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.