dotnet / dotnet/aspnetcore

Microsoft.AspNetCore.Hosting perf counters are slow

Open
#50,412 17 comments 1 reaction 1 assignee Claimed by @JamesNK View on GitHub
area-networking design-proposal Perf
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 10h
Merged PRs (30d)
281

Description

We're trying to benchmark a webapp on a many-cores system and we're mainly interested in RPS metrics at the moment. We don't want to rely on client side RPS metrics (bombardier or Wrk depending on OS) because there are multiple clients and they're not started instantly at the same time. It seems that `Microsoft.AspNetCore.Hosting` perf counters can be used to calculate RPS on aspnet side which is what we need. Unfortunately, it seems that these counters make everything twice slower when enabled.

Basically, it can be simulated on our PerfLab with crank:
```
crank
--config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/json.benchmarks.yml
--config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/azure.profile.yml
--config https://raw.githubusercontent.com/aspnet/Benchmarks/main/scenarios/steadystate.profile.yml
--config https://raw.githubusercontent.com/aspnet/Benchmarks/main/build/ci.profile.yml
--scenario json
--profile arm-lin-28-app
--profile intel-load2-load
--application.framework net8.0
--application.collectDependencies true
--application.options.collectCounters true
--application.options.counterProviders "Microsoft.AspNetCore.Hosting"
```
the last line is the culprit. Is it possible to somehow achieve an overhead-free server-side RPS metric? The current culprit seems to be EvenPipe:

![image](https://github.com/dotnet/aspnetcore/assets/523221/6e9e2a5d-248e-4835-8483-7c3c48350931)

Tried Linux-x64, Linux-arm64 and Windows-x64 machines.

UPD: Current suspects:

| Mode | RPS |
|-------------------------------------------------------------------------------------------------------------------------------------|-------|
| No counters | 1M |
| Collect "System.Runtime" counters | 1M |
| Collect "Microsoft.AspNetCore.Hosting" counters | 0.48M |
| Collect "Microsoft.AspNetCore.Hosting" counters, EventPipeEnableStackwalk=0 | 0.65M |
| Collect "Microsoft.AspNetCore.Hosting" counters, EventPipeEnableStackwalk=0, [without Interlocked](https://github.com/dotnet/aspnetcore/blob/main/src/Hosting/Hosting/src/Internal/HostingEventSource.cs#L55-L64) | 0.75M |
| Collect counters, Microsoft.AspNetCore.Hosting, EventPipeEnableStackwalk=0, without Interlocked, [without lock in CounterAggregator](https://github.com/dotnet/runtime/blob/9c3f8b3727d9be4de483a1d725c2bda22f956688/src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Metrics/CounterAggregator.cs#L19) | 0.8M |

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.