[Tracking] Improve in WASM CI leg throughput

### WASM CI Legs Throughput Investigation

There has been growing demand to improve the throughput of the WASM CI legs, which in recent months have shown slower performance compared to other legs. Several investigations were conducted to identify where time is being spent during execution.

WASM execution has some inherent overhead — it runs inside a browser, relies on interpreters, and operates in a single-threaded environment. Additionally, the infrastructure team identified issues related to queue waiting times, machine provisioning delays, and result reporting latency.

Below is a summary of the investigation findings done by @chcosta:

---

#### Executive Summary

* **Total Duration:** 48 minutes 10 seconds (21:31:32 → 22:19:42 UTC)
* **Queue Delay:** 30 minutes 14 seconds (63% of total time)
* **Execution Time:** 17 minutes 55 seconds (37% of total time)

This confirms earlier assumptions that the primary bottleneck lies in queue delays. While autoscaling may help somewhat, queue capacity and usage require further consideration. Analysis also suggests that increasing parallelization may not yield improvements, as it could lead to additional resource contention.

---

#### Detailed Timeline Breakdown

**Phase 1: Job Creation & Queuing**

* **Duration:** 30m 14s (63%)
* **Details:**

* Job created in Helix and queued on `ubuntu.2204.amd64.open.rt`
* Resource allocation delays on Azure DevOps hosted agents
* Queue contention (203 work items in this job)
* Agent provisioning time

**Phase 2: Job Execution**

* **Duration:** 17m 55s (37%)
* **Highlights:**

* Docker setup using `mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-helix-webassembly`
* Environment setup (56 seconds) including Python, browser initialization, and XHarness setup
* Test discovery (9,935 tests found)
* Test execution (15m 58s): 47,478 tests executed, 47,464 passed, 14 skipped
* Single-threaded browser-based WASM execution through Firefox
* Cleanup and reporting (1m 42s total)

---

#### Performance Analysis

| Phase | Duration | % of Total | Notes |
| ----------------- | -------- | ---------- | -------------------------------------- |
| Queue Time | 30m 14s | 63% | **Primary bottleneck** |
| Environment Setup | 56s | 2% | Minor |
| Test Execution | 15m 58s | 33% | WASM single-threaded overhead |
| Reporting/Cleanup | 1m 42s | 4% | Result upload & Azure DevOps reporting |

---

#### Root Cause Analysis

**Primary Issue:**

* Queue delays exceeding 30 minutes due to high demand on the `ubuntu.2204.amd64.open.rt` queue.
* Large job sizes (200+ work items) competing for limited resources.
* Azure-hosted agent scaling delays.

**Secondary Factors:**

* Single-threaded WASM execution limiting parallelization benefits.
* Browser testing overhead compared to native execution.
* Large result payloads (~18 MB) increasing upload times.

---

#### Additional Observations

* Queue saturation occurred during peak hours, with multiple large browser/WASM jobs running concurrently.
* Estimated 7–12 concurrent jobs with 1,000+ total work items exceeded the 1,200-machine capacity.
* Productive execution accounted for only 38% of total time; 62% was queue waiting.
* Similar job patterns (200+ work items) are common and likely contribute to recurring contention.

---

#### Recommendation

The 48-minute duration falls within the normal range, but queue time remains the dominant factor. Infrastructure improvements targeting queue capacity and scheduling efficiency would yield the most significant performance gains. Autoscaling can help to an extent, but a broader review of queue utilization strategy is recommended.