dotnet / dotnet/msbuild

PerfStar Timing Inconsistencies vs Manual Runs & Unexpected MT Evaluation Behavior (+ OrchardCore on Windows)

Open
#13,766 2 comments 0 reactions 1 assignee Assigned to @AR-May View on GitHub
Area: Performance
Dominant language
C#
Stars
5.5k
Forks
1.5k
Avg merge
1d 8h
Merged PRs (30d)
141

Description

While using PerfStar for performance evaluation of MSBuild scenarios, we are observing inconsistencies in reported timings that reduce confidence in the measurements.
Specifically:

1. Mismatch with manual testing
Timings reported by PerfStar differ from manually executed runs of the same scenarios.
The discrepancy is consistent across multiple runs and scenarios.

2. Unexpected MT (multi-threaded) evaluation performance
Evaluation time in MT mode is higher than in non-MT mode.
This is counterintuitive, as MT is expected to improve or at least not regress performance.
The issue is particularly visible in evaluation phase metrics (as opposed to build phase only).

## Expected Behavior

PerfStar timings should reasonably align with manual measurements (within expected variance).
MT mode should not be consistently slower than non-MT for evaluation.

## Observed Behavior

PerfStar produces timings that diverge from manual runs.
MT evaluation is consistently slower than non-MT in some scenarios.

## Questions

Are PerfStar metrics collected differently than in manual runs?
Are MT scenarios configured identically between PerfStar and manual testing?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.