`externalPackagesPlugin` slows down `ng serve` substantially

### Command

serve

### Is this a regression?

- [x] Kinda, as the `externalPackagesPlugin` was just added via https://github.com/angular/angular-cli/pull/26923

### The previous version in which this bug was not present was

Before https://github.com/angular/angular-cli/pull/26923 existed, which was part of [17.1.1](https://github.com/angular/angular-cli/releases/tag/17.1.1), so I guess angular [17.1.0](https://github.com/angular/angular-cli/releases/tag/17.1.0)

### Description

angular 17.2.0
vite/esbuild powered application builder

Using the `loader` option in `angular.json` like:

```json
"loader": { ".webp": "file", ".svg": "text" },
```

to support images as ecmascript imports or svgs as inline. This causes angular-cli to add the `externalPackagesPlugin` which is a giant hit on build performance. For our apps:

| angular version | watch rebuild time | note |
| -- | -- | -- |
| angular 17.1.0 `{ "loader": { ".webp": "file" } }` | `0.8 seconds` |
| ~angular 17.1.1 `{ "loader": { ".webp": "file" } }`~ | `6.0 seconds` |
| ~angular 17.2.0 `{ "loader": { ".webp": "file" } }`~ | `6.0 seconds` |
| angular 19.2.1 `{ "loader": { ".webp": "file" } }` | `0.8 seconds` | <- `file` doesn't regress anymore |
| angular 19.2.1 `{ "loader": { ".svg": "text" } }` | `6.0 seconds` | <- `text` still does |

Those additional 5.2 seconds are also added to the initial build.

My rough idea of the issue is, that the `externalPackagesPlugin` looks at every single import that doesn't start with `.` or `/` and doesn't cache the results. So tons of `@angular/...`, `rxjs`, `@ngrx/...` and similar imports are resolved over and over again. Even if this function only needs 2ms to run, this quickly adds up.

### Minimal Reproduction

https://github.com/sod/ng-serve-external-deps-slowdown

The only difference between the `development` and `development-slow` configurations in the angular.json are:

```json
"development": {
},
"development-slow": {
"loader": {
".svg": "text"
}
},
```

### Your Environment

```text
@angular-devkit/architect 0.1702.0
@angular-devkit/build-angular 17.2.0
@angular-devkit/core 17.2.0
@angular-devkit/schematics 17.2.0
@angular/cli 17.2.0
@schematics/angular 17.2.0
rxjs 7.8.1
typescript 5.3.3
zone.js 0.14.4
```