Deprecate custom OptProf pipelines
- Dominant language
- C#
- Stars
- 5.5k
- Forks
- 1.5k
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 141
Description
## Context
MSBuild currently owns a custom OptProf/IBC data collection pipeline that:
1. Takes MSBuild bits + VS builds, combines them, and runs OptProf scenarios.
2. Collects profile-guided optimization data and publishes it for later embedding into MSBuild DLLs at build time.
3. Uses Arcade infrastructure to discover the latest drop from an Azure DevOps drop service based on branch name.
### Problems with the Current Approach
- **Fragile infrastructure** — the pipeline uses job definition templates and infrastructure the team has near-zero understanding of. It breaks regularly (most recently, a permissions issue in the VMR broke all VMR builds).
- **Deprecated machine infrastructure** — the data collection runs on infrastructure that is being decommissioned (deadline ~November 2026).
- **VMR coupling** — the VMR must know about inner-repo branch names to pull OptProf data, which violates the VMR's design principle of being branch-agnostic. Hard-coded branch names (e.g., `vs18.4`, `vs18.6`) go stale and cause failures.
- **Questionable ROI** — the team has not observed clear wall-time performance improvements from the embedded data, though VS perf DD rit tests will fail without it (they check JIT counters).
## Proposed Solution
### Phase 1: Let VS own data collection
The VS OptProf team already maintains their own data collection infrastructure with a dedicated team. They are willing to run MSBuild scenarios as part of their collection. This removes the MSBuild team's burden of maintaining the collection pipeline entirely.
### Phase 2: Change data consumption to explicit drops
Instead of dynamically discovering drops by branch prefix at build time (which is fragile), switch to explicit drop references updated periodically (e.g., weekly) via automated PRs. This:
- Eliminates the branch-name coupling in the VMR.
- Makes builds deterministic and not dependent on external service availability.
- Could even be a committed artifact if the data is small enough (it gets embedded in DLLs, so it cannot be very large).
### Alternative: Freeze current data
If the drift from stale OptProf data is slow enough, it may be viable to freeze the current data and never update it. The team observed that even mismatched data (e.g., 1611 data applied to 1714) passes VS perf gates. This could buy years without any infrastructure investment.
### VMR-specific consideration
Desktop MSBuild assemblies are not shipped in the .NET SDK. It may be possible to disable OptProf consumption in the VMR entirely, which would immediately eliminate VMR-side pain.
## Exit Criteria
- MSBuild no longer owns or maintains an OptProf data collection pipeline.
- OptProf data consumption is reliable and does not cause build failures due to service availability issues.
- VS perf DD rit tests continue to pass.
## Next Steps
1. Attend VS Perf team office hours to understand the concrete steps for Phase 1.
2. Estimate effort and decide between Phase 2 vs. freezing data.
3. Investigate disabling OptProf in the VMR as a quick win.
## Notes
- Discussed in grooming on June 2 and June 10, 2026.
- The infrastructure decommission deadline is approximately November 2026.
- An experimental insertion without OptProf data was discussed to validate whether VS perf gates actually fail.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.