Performance regression in #33473

The extra work done in #33473 is valuable, but maybe it doesn't need to be so expensive. In this particular case, there don't seem to be any errors (though there is one @ts-ignore).

Setup:

1. Clone https://github.com/amcasey/material-ui.git
2. Check out Benchmark
3. yarn to restore packages
4. yarn typescript to prebuild and run some TS tests

Repro

1. tsc -p docs

On a random Mac Mini:

• 10-run avg on https://github.com/microsoft/TypeScript/commit/6c2ae12559508b594f7038f8d6165b92ec89c58c: 33,010ms
• 10-run avg on https://github.com/microsoft/TypeScript/commit/26caa3793e310e271ddee8adc1804486e5b0749f: 33,494ms (1% slower)

Note: to build old TS commits, you probably need to change const { default: chalk } to const chalk in scripts/build/utils.js.