rust-lang / rust-lang/rust

Compiler takes 30 minutes + large (32GB+) amounts of ram to compile code with large amount of arguments + function-within-function

Open
#122,944 21 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-macros C-bug E-needs-mcve I-compilemem I-slow
Dominant language
Rust
Stars
119k
Forks
16.1k
PR merge metrics
PR metrics pending

Description

I do not have a simple reproducible example since this bug goes over my head, perhaps someone else can find the bug but it seems pretty bad:

This file takes over 30 minutes of time to compile in rustc 1.77.0 (rustc 1.77.0 (aedd173a2 2024-03-17)) and rustc nightly (rustc 1.78.0-nightly (2bf78d12d 2024-02-18)): https://github.com/Anti-Raid/splashtail/blob/74a33ddebe045d2a9fb4acc2d65ef97f75314e84/botv2/src/modules/moderation/cmd.rs [this specific commit triggered the issue of 30 minutes though this was always exponentially slow from the very start]

Don't know why it is, but this seemingly innocent file takes up a full 25-30 minutes of compile time and hogs up over 32GB of ram to the point that I needed to add 128GB of swap in order for Linux's Out Of Memory killer to not SIGKILL rust. In CI, this just leads to a SIGKILL quickly within 20 minutes of compiling. Actual compilation seems to go through without much of an error. Removing moderation from the module tree reduces compile times from 30 minutes to 5-6 minutes and RAM usage returns to normal

I expected to see this happen: Compile times within 5-6 minutes (which is normal right now, don't know if the 5 minute time comes from the same root cause as this bug though but it could be) and normal RAM usage (<32GB ram)

Instead, this happened:

warning: struct ConfigOption is never constructed
--> src/silverpelt/config_opt.rs:159:12
|
159 | pub struct ConfigOption {
| ^^^^^^^^^^^^
|
= note: #[warn(dead_code)] on by default

warning: botv2 (bin "botv2") generated 1 warning
Finished release [optimized] target(s) in 30m 27s

[Ignore the warning, thats unrelated]

Linux kernel kills rust with a SIGKILL 9 randomly and Out Of Memory killer activates

Meta

Tested and reproduced across both Apple M1 and a Dedicated Server (Intel(R) Xeon(R) CPU E3-1240 v3 @ 3.40GHz) on both rust stable and nightly using both LLVM and Cranelift as codegen backends. Switching linkers to clang+mold/ld.lld from defaults did not help. Rust versions are given below:

rustc 1.77.0 (aedd173a2 2024-03-17) and rustc 1.78.0-nightly (2bf78d12d 2024-02-18)

No backtrace is available, the compiler does not crash, attempts to compile the code using --timing for timing information don't seem to even complete in any reasonable timespan. Issue occurs on both debug and release builds so this is not an issue of release build optimizations. Typical options to speed up compile times such as opt-level = 1 do not work in reducing neither the compile times nor the memory usage. Splitting up the code into multiple crates does not work as long as this file is in the module tree and hence compiled.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with botv2/src/modules/moderation/cmd.rs at the linked commit and reproduce the build with rustc 1.77.0 and the noted nightly version. Compare compilation with and without moderation in the module tree, and record timing and memory usage; the issue is complete when the expensive construct is isolated in a smaller reproduction or its compiler cause is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.