Insanely slow response times when using ADC auth and not regular auth

### Please make sure you have searched for information in the following guides.

- [x] Search the issues already opened: https://github.com/GoogleCloudPlatform/google-cloud-node/issues
- [x] Search StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js
- [x] Check our Troubleshooting guide: https://github.com/googleapis/google-cloud-node/blob/main/docs/troubleshooting.md
- [x] Check our FAQ: https://github.com/googleapis/google-cloud-node/blob/main/docs/faq.md
- [x] Check our libraries HOW-TO: https://github.com/googleapis/gax-nodejs/blob/main/client-libraries.md
- [x] Check out our authentication guide: https://github.com/googleapis/google-auth-library-nodejs
- [x] Check out handwritten samples for many of our APIs: https://github.com/GoogleCloudPlatform/nodejs-docs-samples

### A screenshot that you have tested with "Try this API".

`time gsutil ls gs://gcp-public-data-arco-era5/ar/model-level-1h-0p25deg.zarr-v1/hybrid` (example bucket) gives this:
```
real 0m1.896s
user 0m0.383s
sys 0m0.152s
```

### Link to the code that reproduces this issue. A link to a **public** Github Repository or gist with a minimal reproduction.

https://github.com/AndyClausen/storage-performance-issue-example

### A step-by-step description of how to reproduce the issue, based on the linked reproduction.

Scenario 1 - requests hang forever:
1. run `gcloud auth revoke` to make sure you're not logged into gcloud normally
2. run `gcloud auth application-default login` and log in
3. run `bun i` to install deps
4. run `bun start`

Scenario 2 - everything works:
1. run `gcloud auth login` and log in
2. run `gcloud auth application-default login` and log in
3. run `bun i` to install deps
4. run `bun start`

Scenario 3 - requests hang for about 15 seconds:
1. run `gcloud auth login` using an account that is set up to require re-authentication after a period of time
2. wait for said amount of time
3. run `gcloud auth application-default login` and log in
4. run `bun i` to install deps
5. run `bun start`

### A clear and concise description of what the bug is, and what you expected to happen.

There's something going on with authentication. If I log in to gcloud with ADC only, requests hang seemingly forever. If I log in with regular auth and ADC, everything works fine and it uses ADC. If I log in with regular auth and wait for the token to expire, and log in with ADC freshly, it hangs for around 15 seconds.

It's a bit difficult to reproduce the last scenario, but the first two should be simple.

The reproduction repo uses `getFiles` as an example, and the example bucket is just some public bucket I found from a quick google search.

### A clear and concise description WHY you expect this behavior, i.e., was it a recent change, there is documentation that points to this behavior, etc. **

I would expect the library to ignore regular auth and just use ADC, but it seems like it's checking regular auth credentials first for some reason, even though it ends up just using ADC. This is not an issue with gsutil, so I don't know why it is with this library.