fastbuild has become slow

I'm seeing a 5x slowdown building my crate with #1091.

Building with latest main:

> 38.153 s 92.26% action 'Compiling Rust rlib anki (198 files)'

After reverting #1091:

> 7.646 s 83.10% action 'Compiling Rust rlib anki (198 files)'

Building my crate in release mode is actually faster than opt level 1:

> 26.012 s 92.79% action 'Compiling Rust rlib anki (198 files)'

Perhaps my crate is an outlier here, but I thought I should mention it, as it's more than "very slightly" :-)