Seeking .NET Runtime Team's experience and advice on running benchmarks in Azure AKS

Hello,

We are currently running CPU-intensive benchmarks using BenchmarkDotNet v0.13.0 inside containers deployed on Azure AKS (Linux). These benchmarks make heavy use of hardware intrinsics (e.g., AVX2, AVX-512), and some of them also involve moderate I/O operations (e.g, file access).

We are trying to identify best practices for running these benchmarks reliably and reproducibly in a cloud-native environment, and we would like to learn from your experience.

We are seeking guidance on the following:

- **Do you use Azure VMs for your own performance or BenchmarkDotNet tests ?**
If yes, which VM SKUs do you typically choose for workloads that rely heavily on CPU intrinsics and some I/O?

- **Any guidance or lessons learned?**
For example: How do you handle variability across VM instances ? Recommended iteration settings (`IterationCount`, `WarmupCount`) for convergence in cloud environment.

- **Comparison strategies**
We currently store a fixed reference benchmark in our Git repository and update it only when differences are justified.
We are also considering comparing performance before and after a commit when code changes occur.
Do you have best practices or tooling suggestions for these approaches?

- **Cloud-specific issues**
Have issues like noisy neighbors, CPU frequency scaling, or virtualization overhead been addressed in your benchmarking process? Any configuration tweaks you recommend?

- **.NET Runtime configuration**
Are there recommended runtime settings for accurate benchmarking in cloud environments? For example, should we explicitly configure GC mode (Server vs Workstation) or tweak JIT settings to reduce variability?

- **Accepted variance between runs**
What is a reasonable threshold for performance deviation between two benchmark runs in a cloud environment? For example, is +/- 10% considered acceptable, or should we aim for tighter bounds?

I have also published a related [question](https://github.com/dotnet/BenchmarkDotNet/issues/2844) in the BenchmarkDotNet GitHub repository for community input.

**Current Setup**
- Benchmarks are run in containers based on .NET 8.0.
- Pods are scheduled on dedicated AKS node pools (currently testing AMD [Dadsv6](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/general-purpose/dadsv6-series?tabs=sizebasic)).
- Some benchmarks include I/O operations (e.g., reading/writing files).
- We are using version v0.13.0 of BenchmarkDotNet.

If sharing details here is not possible, we can open a support ticket instead. Just let us know what works best.

Any insights, recommendations, or community experiences would be greatly appreciated!

Thanks