rust-lang / rust-lang/rust

bug: LLVM22 upgrade causes severe compile time regression

Open
#157,302 5 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-LLVM C-bug I-compiletime I-prioritize regression-from-stable-to-stable T-compiler
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

During work on https://github.com/zama-ai/kms we noticed that upgrading the toolchain from 1.94 to 1.95 caused compile times to balloon from 12min to 2h30m. We set it aside and then tried again with 1.96: same result.

The investigation shows that compilation gets stuck on one of our dependencies, tfhe-fft . The problem is specific to Linux/x86, where the AVX-512/pulp codegen path causes a ~45x slowdown; running with --timings confirms that this is a codegen issue, so distinct from the Bevy's issue with the frontend.

Looking at the compilation with perf top hints at the SLP vectorizer, but that's where our investigation ended.

Code

This is the most minimal repro we've been able to cook up. Run cargo new slp-repro and replace Cargo.toml with:

[package]
name = "slp-repro"
version = "0.1.0"
edition = "2021"

[dependencies]
tfhe-fft = { version = "=0.10.1", features = ["fft128", "serde"] }

# tfhe-fft must be optimized for the SLP blowup to appear. Keep the dev profile
# (debug-info + assertions on); a `--release` build does NOT reproduce.
[profile.dev]
opt-level = 3

Leave src/main.rs as the generated stub. Then:

$ cargo +1.94.0 build      # LLVM 21 — finishes in ~5 minutes
$ cargo +1.96.0 build      # LLVM 22 — takes hours

[!IMPORTANT]

  • x86_64 only. The blowup is in the AVX-512 path (tfhe-fft's avx512 feature → pulp/x86-v4). On aarch64 that path is scalar and there is no regression.
  • Build in dev (cargo build), not --release. A --release build is fast (because it strips the optimize-with-debug-info combination?); the regression needs tfhe-fft at opt-level = 3 under the dev profile.
Version it regressed in

First shipped in stable 1.95.0 (the first release with LLVM 22); also present in 1.96.0 and every nightly since.

1.94.0 (LLVM 21): good
1.95.0 (LLVM 22): bad   (untested directly, but contains the regressing commit)
1.96.0 (LLVM 22): bad
Regression bisection

Bisected with cargo-bisect-rustc (predicate = "tfhe-fft build exceeds 600s"):

searched nightlies: from nightly-2026-01-22 to nightly-2026-01-29
regressed nightly:  nightly-2026-01-29
regressed commit:   466ea4e6c39f8a43727edcc726ca86b499e14d83

That commit is #150722 ("Update to LLVM 22", rustc/21.1rustc/22.1-2026-01-27) — a pure LLVM submodule bump, no rustc codegen changes.

the SLP vectorizer?

Toggling only -C llvm-args=-vectorize-slp=false, with everything else fixed, isolates the cost to the SLP vectorizer. Cold cargo build of tfhe-fft, wall-clock:

LLVM 21 (1.94.0) LLVM 22 (1.96.0)
SLP on (default) 286s ~9200s
SLP off (-vectorize-slp=false) 145s 199s

So ≈99% of the regression is the SLP vectorizer. With SLP disabled, the LLVM-22 build (199s) is faster than the LLVM-21 build (286s).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the minimal Cargo.toml using tfhe-fft 0.10.1 and compare cargo +1.94.0 build with cargo +1.96.0 build on Linux/x86_64. Use --timings, perf top, and -C llvm-args=-vectorize-slp=false to confirm the SLP vectorizer regression, then inspect LLVM commit 466ea4e6c39f8a43727edcc726ca86b499e14d83. Done means the LLVM 22 toolchain no longer exhibits the hours-long dev-profile build.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.