Optimize TypeSpec Validation CI runtime in the spec repo

## Description:

The **TypeSpec Validation CI** in the spec repo is taking too long to complete. In many cases, the run time is around an hour or even longer, which significantly slows down development and validation workflows.

Example link: https://github.com/Azure/azure-rest-api-specs/actions/runs/28857085073/job/85586200452?pr=44494

## Expected behavior:
TypeSpec Validation CI should finish in a reasonable amount of time, ideally much faster than the current runtime.

## Actual behavior:
The CI often takes one hour or more to complete.

## Impact:

- Slower feedback loop for spec changes
- Reduced development efficiency
- Longer wait time for validation results

@mikeharder, @qiaozha for notification.

Start by inspecting the TypeSpec Validation CI and the linked GitHub Actions run to identify which jobs or steps account for the long runtime. Done means the validation workflow completes substantially faster while preserving its validation coverage; the issue does not name specific workflow files or tests to run.

Written by the indexing model from the issue text.