Python: Sample validation reports Copilot infrastructure failures as sample regressions

Description

The scheduled Python - Sample Validation workflow currently mixes sample regressions with failures in the Copilot-based validator itself.

Ten validator jobs run concurrently. Each invocation uses the CLI default of ten in-flight workers, so there is no workflow-wide limit preventing roughly one hundred Copilot validation requests from starting together.

The worker then retries every exception once by immediately restarting the agent. It does not delay or distinguish transient provider failures. If the retry also fails, the exception is recorded as an ordinary sample FAILURE.

Current implementation:

• --max-parallel-workers defaults to 10
• each validator creates and dispatches that many Copilot workers
• all exceptions get one immediate restart/retry and then become RunStatus.FAILURE

The latest checked run, 29622990122, has multiple validator jobs reporting errors such as:

Error: 429 "too many requests"
You've hit your rate limit. Please wait for your limit to reset in under a minute
Timeout after 120s waiting for session.idle

Examples include OpenAI provider validation, GitHub Copilot provider validation, and Semantic Kernel migration validation. The previous scheduled run, 29544765836, shows the same pattern.

The latest 100 scheduled workflow runs, from April 10 through July 18, all concluded with failure. That does not mean every run failed only because of quota exhaustion, but the current classification makes the trend unable to distinguish a real sample regression from validator infrastructure noise.

Expected behavior

• Bound aggregate Copilot concurrency across the scheduled workflow, rather than applying a per-job limit only.
• Retry recognized transient infrastructure errors with an appropriate delay, jitter, and provider retry guidance where available.
• Report validator infrastructure failures separately from sample failure and missing_setup, so they do not appear as regressions in the sample trend.
• Continue to fail or otherwise surface the workflow when validation infrastructure is unavailable, but preserve the reason accurately.

Closed draft PR #6747 explored broader workflow changes and temporarily disabled two jobs to free quota, but it did not add aggregate throttling, delayed transient retries, or separate infrastructure classification.

If this direction is acceptable, I can take the focused harness fix after agreement and add regression coverage for transient retry and result classification.