NVIDIA / NVIDIA/cccl

Profile cmake generation

Open
#7,298 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
2.5k
Forks
486
Avg merge
2d 6h
Merged PRs (30d)
295

Description

I recently learned you can profile cmake using `--profiling-format=google-trace --profiling-output=cmake_profile.json`. Since running cmake is not fast, I wondered what we can find. CMake configure time matters especially on clusters with network file systems, were cmake generation can take a couple of minutes.

So I tried it using a cmake command I commonly use to get the `all-dev` preset:

```bash
cccl/build_prof$ CXX=clang++-20 cmake .. --preset all-dev -B. -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_CUDA_HOST_COMPILER=clang++-20 -DTHRUST_ENABLE_MULTICONFIG=OFF -DCCCL_ENABLE_BENCHMARKS=OFF -DCMAKE_CUDA_FLAGS="--ftemplate-backtrace-limit=0 -lineinfo" -DCMAKE_CUDA_ARCHITECTURES=120 --profiling-format=google-trace --profiling-output=cmake_profile.json
```
results in this graph:
Image
We can see that fetching dependencies via `fetchcontent` is expensive.

On a second run of the above command, where we already have an existing binary directory with fetched dependencies:
Image
We now see that generating the build system takes the most time. I assume because we have a LOT of targets.

This is for CUB tuning, which we run very frequently on clusters:
```
cccl/build_prof$ cmake .. --preset=cub-tune
```
Image
We can see that `try_compile` from CMake takes some time (I guess we cannot change that), and the other large block is getting nvbench, which pulls in catch2 (which we wouldn't need as consumers of nvbench), and libfmt and nlohmann_json (both via rapids_cpm_find).

We could consider not fetching Catch2 when nvbench is consumed as sub project, and refactoring libfmt to `std::format` so we can lose the dependency.

In any case, profiling cmake is surprisingly easy and insightful and there are probably some low hanging fruits on how to configure faster that we could try.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.