Some rules unnecessarily slow because of unneeded creation of import map

Hi,

While investigating the performance of our linter setup, I ran a TIMING=1 type performance scan of my linter rules, and found that unfortunately, when any of the used rules needs an import map, it will look like the first rule who uses it, takes 90+% of the work. Example:

> DEBUG=eslint:cli-engine TIMING=1 yarn lint --debug --verbose

All files pass linting.

Rule                              | Time (ms) | Relative
:---------------------------------|----------:|--------:
import/namespace                  | 10247.586 |    98.0%
import/no-cycle                   |   114.264 |     1.1%
@nx/enforce-module-boundaries     |    30.531 |     0.3%
import/no-unresolved              |    10.439 |     0.1%
@typescript-eslint/no-unused-vars |     6.393 |     0.1%

eslint-plugin-import seems to suffer from this most, in fact, when I disable all import/* rules, the above issue goes away: the time spent between rules is more balanced. Example:

Rule                              | Time (ms) | Relative
:---------------------------------|----------:|--------:
@nx/enforce-module-boundaries     |    16.948 |    22.5%
@typescript-eslint/no-unused-vars |    13.912 |    18.4%
jest/no-duplicate-hooks           |     2.725 |     3.6%
sonarjs/cognitive-complexity      |     1.869 |     2.5%
sonarjs/no-gratuitous-expressions |     1.668 |     2.2%

Discussing with the ESLint maintainers how I might overcome the misbalance in reported time spent, one notable statement came back regarding eslint-plugin-import:

Overall, eslint-plugin-import rules always try to resolve every import even when they don't need to (looking at you, no-unused-imports), and that is, in fact, what makes them slow. The maintainers seem to think that other plugins/rules will prepopulate the import map for them, but that's just not the case in the current ecosystem.

Would you say this is accurate? Could the the rules who don't need the import map, be made not to trigger the building of one? Or the other way around? Granted I don't fully understand the implication, or its implications.

Reproduce the reported TIMING=1 yarn lint --debug --verbose comparison and trace how eslint-plugin-import rules trigger creation of the import map. No source files or tests are named; done should establish which rules need the map and prevent unnecessary construction without changing rule behavior.