bazelbuild / bazelbuild/rules_apple

Add support to use shared clangstat cache

Open
#1,984 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Starlark
Stars
593
Forks
334
Avg merge
16h 48m
Merged PRs (30d)
9

Description

**Context**

Source: https://reviews.llvm.org/D136651

```
Every Clang instance uses an internal FileSystemStatCache to avoid
stating the same content multiple times. However, different instances
of Clang will contend for filesystem access for their initial stats
during HeaderSearch or module validation.

On some workloads, the time spent in the kernel in these concurrent
stat calls has been measured to be over 20% of the overall compilation
time. This is extremly wassteful when most of the stat calls target
mostly immutable content like a SDK.

This commit introduces a new tool `clang-stat-cache` able to generate
an OnDiskHashmap containing the stat data for a given filesystem
hierarchy.

The driver part of this has been modeled after -ivfsoverlay given
the similarities with what it influences. It introduces a new
-ivfsstatcache driver option to instruct Clang to use a stat cache
generated by `clang-stat-cache`. These stat caches are inserted at
the bottom of the VFS stack (right above the real filesystem).
```

**Generating the stat cache (as done by Xcode builds)**
$DEVELOPER_DIR/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang-stat-cache $DEVELOPER_DIR/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator16.4.sdk -o /Users/chirag.ramani/Library/Developer/Xcode/DerivedData/SDKStatCaches.noindex/iphonesimulator16.4-20E238-.sdkstatcache

Note: this is specific to the target architecture.

**Using the stat cache (as done by Xcode builds)**
-Xcc -ivfsstatcache -Xcc ~DerivedData/SDKStatCaches.noindex/iphonesimulator16.4-20E238-.sdkstatcache

**Benchmarks of one of our root targets**

**Benchmark 1: ./with_stat_cache.sh**
Time (mean ± σ): 57.663 s ± 1.188 s [User: 32.640 s, System: 23.834 s]
Range (min … max): 56.587 s … 58.939 s 3 runs

**Benchmark 1: ./without_stat_cache.sh**
Time (mean ± σ): 60.126 s ± 1.123 s [User: 33.393 s, System: 26.118 s]
Range (min … max): 59.408 s … 61.420 s 3 runs

We don't see a significant improvement but still seems like a nice to have thing for all ObjC and Swift compilations.

Contributor guide

Open the contributing guide

Research direction

Start by locating the Apple compiler action or toolchain configuration that assembles Clang flags, then compare its existing VFS-related handling with the documented -ivfsstatcache and clang-stat-cache entry points. Confirm the expected generated-cache path and target-architecture behavior from the issue, and validate completion with an Apple compilation using the shared cache.

Written by the indexing model from the issue text.

Assessment

Domain
build-system
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.