llvm / llvm/llvm-project

[libc++] Improve story for CI testing libc++ against Clang trunk changes

Open
#211,893 7 comments 1 reaction 1 assignee Claimed by @ldionne View on GitHub
infrastructure libc++ test-suite
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

Right now, the libc++ CI tests against a nightly build of Clang obtained from Compiler Explorer. That version of Clang gets baked into our Docker image once and then we use the same Clang version until the image gets updated (with a newer nightly snapshot).

This creates a difficulty when we make changes to Clang and want to update libc++ to rely on those. Indeed, imagine Clang 24 is the current "in development" version. In the CI image, we have a snapshot of Clang trunk (which reports itself as clang-24) built at some point, let's say yesterday.

Now, we land a change to Clang which adds e.g. a new builtin, and want to make use of it in the libc++ code, and test that. Using the feature in the code is usually fairly easy because we can guard that usage on something provided by Clang itself like `__has_builtin` or similar, and that allows us to faithfully detect whether the thing we need is available. However, from the test suite, we rely on version checks like `// UNSUPPORTED: clang-23`. Now, the problem is that we can't actually write a test that relies on the feature since the snapshot of Clang trunk we have in the Docker image was built previously, and it doesn't support the new feature. In other words, we end up having to use `// UNSUPPORTED: clang-22, clang-23, clang-24` even though a recent-enough version of Clang 24 would support the feature.

That makes it extremely difficult to test anything that relies on recent Clang changes in the libc++ CI. To address this, one option would be to start testing against the just-built Clang in libc++'s CI pipeline instead. That way, changes made to Clang would be picked up immediately in libc++'s CI, and `clang-24` in the test suite would refer to the true latest version of Clang, not some snapshot that's a few days/weeks old.

Possible CI structure if we do this:
- Build Clang from source as the first thing in the libc++ CI
- Upload it as an artifact
- Then from libc++ CI jobs, use that build artifact instead of the compiler explorer snapshot
- Note that we'd still keep jobs for previously-released versions of Clang in our matrix, since we still support them. But the majority of testing (that we currently do against the CE snapshot) would now move to being against a just-built Clang.

This raises some questions:
- How costly is it? Can we afford to actually build Clang from scratch before we start other libc++ jobs? I think this needs to be tested.
- How often is Clang ToT broken, and for how long? Testing libc++ against just-built Clang implies that libc++ is now vulnerable to any change that breaks Clang. Right now, using a snapshot has the downside that it's often stale, but at least it's stable.
- Can we add something to the Clang CI itself to ensure that our workflow doesn't get broken? In other words, is the current Clang CI enough to ensure that libc++'s CI stays green? I think the minimum here would be to ensure that we build the libc++ test suite on Clang changes. We probably can't afford to run our whole 70+ configurations CI pipeline on every Clang change, but we could perhaps run one or two common libc++ configurations on every Clang change. Then, assuming Clang keeps that CI green 100% of the time, libc++'s own CI would stay green as well (except for things that are only caught by configurations not tested in the Clang CI, but I presume that would be negligible).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.