PerfStar Timing Inconsistencies vs Manual Runs & Unexpected MT Evaluation Behavior (+ OrchardCore on Windows)
- Dominant language
- C#
- Stars
- 5.5k
- Forks
- 1.5k
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 141
Description
While using PerfStar for performance evaluation of MSBuild scenarios, we are observing inconsistencies in reported timings that reduce confidence in the measurements.
Specifically:
1. Mismatch with manual testing
Timings reported by PerfStar differ from manually executed runs of the same scenarios.
The discrepancy is consistent across multiple runs and scenarios.
2. Unexpected MT (multi-threaded) evaluation performance
Evaluation time in MT mode is higher than in non-MT mode.
This is counterintuitive, as MT is expected to improve or at least not regress performance.
The issue is particularly visible in evaluation phase metrics (as opposed to build phase only).
## Expected Behavior
PerfStar timings should reasonably align with manual measurements (within expected variance).
MT mode should not be consistently slower than non-MT for evaluation.
## Observed Behavior
PerfStar produces timings that diverge from manual runs.
MT evaluation is consistently slower than non-MT in some scenarios.
## Questions
Are PerfStar metrics collected differently than in manual runs?
Are MT scenarios configured identically between PerfStar and manual testing?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.