NVIDIA / NVIDIA/cccl

[BUG]: devcontainers failing to compile program.

Open
#593 8 comments 2 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.5k
Forks
487
Avg merge
2d 7h
Merged PRs (30d)
296

Description

### Is this a duplicate?

- [X] I confirmed there appear to be no [duplicate issues](https://github.com/NVIDIA/cccl/issues) for this bug and that I agree to the [Code of Conduct](CODE_OF_CONDUCT.md)

### Type of Bug

Compile-time Error

### Component

Not sure

### Describe the bug

The devcontainers currently rely on `sccache`. For people who are not employees of NVIDIA you will not be able to rely on the cloud so obviously sccache should default to the home directory. However with the build scripts, what happens is that `sccache` will try to remotely connect to the server, fail, and then stop compiling.

Here's a modification to the `devcontainer.json` I did that fixes the problem, but this isn't a permanent solution and is more something I did on my end because I just wanted to get set up right away.

```{
"shutdownAction": "stopContainer",
"image": "rapidsai/devcontainers:23.12-cpp-gcc12-cuda12.2-ubuntu22.04",
"hostRequirements": {
"gpu": true
},
"initializeCommand": [
"/bin/bash",
"-c",
"mkdir -m 0755 -p ${localWorkspaceFolder}/.{cache,config,sccache}"
],
"containerEnv": {
"SCCACHE_DIR": "${containerWorkspaceFolder}/.sccache",
"HISTFILE": "${containerWorkspaceFolder}/.cache/._bash_history",
"DEVCONTAINER_NAME": "cuda12.2-gcc12",
"CCCL_CUDA_VERSION": "12.2",
"CCCL_HOST_COMPILER": "gcc",
"CCCL_HOST_COMPILER_VERSION": "12",
"CCCL_BUILD_INFIX": "cuda12.2-gcc12",
"CXX": "g++",
"CUDAHOSTCXX": "g++"
},
"workspaceFolder": "/home/coder/${localWorkspaceFolderBasename}",
"workspaceMount": "source=${localWorkspaceFolder},target=/home/coder/${localWorkspaceFolderBasename},type=bind,consistency=consistent",
"mounts": [
"source=${localWorkspaceFolder}/.cache,target=/home/coder/.cache,type=bind,consistency=consistent",
"source=${localWorkspaceFolder}/.config,target=/home/coder/.config,type=bind,consistency=consistent"
],
"customizations": {
"vscode": {
"extensions": [
"llvm-vs-code-extensions.vscode-clangd",
"xaver.clang-format"
],
"settings": {
"editor.defaultFormatter": "xaver.clang-format",
"clang-format.executable": "/usr/local/bin/clang-format",
"clangd.arguments": [
"--compile-commands-dir=${workspaceFolder}"
]
}
}
},
"name": "cuda12.2-gcc12"
}
```
As you can see I am removing the cloud from the picture and just configuring it to my local.

### How to Reproduce

1. Try to compile with a dev container while not connected to any of NVIDIA's access packages.

### Expected behavior

```
CMake Error at /usr/share/cmake-3.27/Modules/CMakeTestCXXCompiler.cmake:60 (message):
The C++ compiler

"/usr/bin/gcc"

is not able to compile a simple test program.

It fails with the following output:

Change Dir: '/home/coder/cccl/build/cuda12.2-llvm16/cub-cpp17/CMakeFiles/CMakeScratch/TryCompile-Us9ZRD'

Run Build Command(s): /usr/local/bin/ninja -v cmTC_5f59e
[1/2] /usr/bin/sccache /usr/bin/gcc -o CMakeFiles/cmTC_5f59e.dir/testCXXCompiler.cxx.o -c /home/coder/cccl/build/cuda12.2-llvm16/cub-cpp17/CMakeFiles/CMakeScratch/TryCompile-Us9ZRD/testCXXCompiler.cxx
FAILED: CMakeFiles/cmTC_5f59e.dir/testCXXCompiler.cxx.o
/usr/bin/sccache /usr/bin/gcc -o CMakeFiles/cmTC_5f59e.dir/testCXXCompiler.cxx.o -c /home/coder/cccl/build/cuda12.2-llvm16/cub-cpp17/CMakeFiles/CMakeScratch/TryCompile-Us9ZRD/testCXXCompiler.cxx
sccache: error: Timed out waiting for server startup. Maybe the remote service is unreachable?
Run with SCCACHE_LOG=debug SCCACHE_NO_DAEMON=1 to get more information
ninja: build stopped: subcommand failed.

CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
CMakeLists.txt:17 (project)
```

And when you run with
`SCCACHE_LOG=debug SCCACHE_NO_DAEMON=1` the following output appears:

```
"/home/coder/.config/sccache/config"
[2023-10-22T16:12:45Z DEBUG sccache::config] Couldn't open config file: failed to open file `/home/coder/.config/sccache/config`
[2023-10-22T16:12:45Z DEBUG sccache::config] Attempting to read config file at "/home/coder/.config/sccache/config"
[2023-10-22T16:12:45Z DEBUG sccache::config] Couldn't open config file: failed to open file `/home/coder/.config/sccache/config`
[2023-10-22T16:12:45Z INFO sccache::server] start_server: port: 4226
[2023-10-22T16:12:45Z INFO sccache::server] No scheduler address configured, disabling distributed sccache
[2023-10-22T16:12:45Z DEBUG sccache::cache::cache] Init s3 cache with bucket rapids-sccache-devs, endpoint None
[2023-10-22T16:12:45Z DEBUG opendal::services::s3::backend] backend build started: Builder { root: None, bucket: "rapids-sccache-devs", endpoint: None, region: Some("us-east-2"), .. }
[2023-10-22T16:12:45Z DEBUG opendal::services::s3::backend] backend use root /
[2023-10-22T16:12:45Z DEBUG opendal::services::s3::backend] backend use bucket rapids-sccache-devs
[2023-10-22T16:12:45Z DEBUG reqsign::aws::config] load_via_profile_config_file failed: No such file or directory (os error 2)

Stack backtrace:
0:
1:
2:
3:
4:
5:
6:
7:
[2023-10-22T16:12:45Z DEBUG reqsign::aws::config] load_via_profile_shared_credentials_file failed: No such file or directory (os error 2)

Stack backtrace:
0:
1:
2:
3:
4:
5:
6:
7:
[2023-10-22T16:12:45Z DEBUG opendal::services::s3::backend] backend use region: us-east-2
[2023-10-22T16:12:45Z DEBUG opendal::services::s3::backend] backend use endpoint: https://s3.us-east-2.amazonaws.com/rapids-sccache-devs
[2023-10-22T16:12:45Z DEBUG opendal::services::s3::backend] backend build finished
[2023-10-22T16:12:45Z DEBUG opendal::services] service=s3 operation=metadata -> started
[2023-10-22T16:12:45Z DEBUG opendal::services] service=s3 operation=metadata -> finished: AccessorInfo { scheme: S3, root: "/", name: "rapids-sccache-devs", capability: { Stat | Read | Write | CreateDir | Delete | Copy | List | Presign | Batch } }
[2023-10-22T16:12:45Z DEBUG opendal::services] service=s3 operation=read path=.sccache_check range=0- -> started
[2023-10-22T16:12:45Z DEBUG reqwest::connect] starting new connection: http://169.254.169.254/
[2023-10-22T16:12:45Z DEBUG hyper::client::connect::http] connecting to 169.254.169.254:80
sccache: error: Timed out waiting for server startup. Maybe the remote service is unreachable?
Run with SCCACHE_LOG=debug SCCACHE_NO_DAEMON=1 to get more information
[2023-10-22T16:14:58Z DEBUG reqsign::aws::credential] load credential via imds_v2 failed: error sending request for url (http://169.254.169.254/latest/api/token): error trying to connect: tcp connect error: Operation timed out (os error 110)

Caused by:
0: error trying to connect: tcp connect error: Operation timed out (os error 110)
1: tcp connect error: Operation timed out (os error 110)
2: Operation timed out (os error 110)

Stack backtrace:
0:
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
[2023-10-22T16:14:58Z WARN opendal::services] service=s3 operation=read path=.sccache_check range=0- -> errored: PermissionDenied (temporary) at read => no valid credential found, please check configuration or try again

Context:
service: s3
path: .sccache_check
range: 0-

[2023-10-22T16:14:58Z ERROR sccache::server] storage check failed for: cache storage failed to read: PermissionDenied (temporary) at read => no valid credential found, please check configuration or try again

Context:
service: s3
path: .sccache_check
range: 0-


Stack backtrace:
0:
1:
2:
3:
4:
5:
6:
7:
[2023-10-22T16:14:58Z DEBUG sccache::server] notify_server_startup(Err { reason: "cache storage failed to read: PermissionDenied (temporary) at read => no valid credential found, please check configuration or try again\n\nContext:\n service: s3\n path: .sccache_check\n range: 0-\n" })
sccache: error: No such file or directory (os error 2)
ninja: build stopped: subcommand failed.
```

### Reproduction link

_No response_

### Operating System

all dev containers

### nvidia-smi output

_No response_

### NVCC version

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.