Significant lock contention with parallel JIT compilation via expression trees

Open
#107,197 11 comments 0 reactions 1 assignee View on GitHub

@EgorBo is already working on this.

Since Sep 3, 2024.

Assessment

This issue has not been assessed yet.

Description

area-CodeGen-coreclr tenet-performance
Description

When doing a lot of JIT compilation in parallel, there seems to be heavy contention surrounding CEEInfo::reportInliningDecision. I managed to trigger this through some heavy usage of expression tree compilation we have in our project.

I was trying to parallelize our integration tests, because for some reason somebody had put them to be limited to 3x parallelism. I removed that cap so the tests could go run at x24 on my machine, and found that they wouldn't progress at all. Digging into this with Rider's timeline profiler I found the following:

Almost all the CPU time in the parallel test run was being used by parallel expression tree compilation, and almost all of that was spent inside CEEInfo::reportInliningDecision blocking on a critical section. Screenshot of the profile I made down below.

I need to make it clear that we are effectively JITing some 20k+ dynamicmethods, times 24, in parallel here. This is due to how our integration tests and serialization system works (yeah, I've been meaning to fix our serialization system, I know). When just starting our game in normal conditions we still see significant amounts of CPU time in reportInliningInfo, but it's dwarfed by unwind info reporting from the profiler being attached. Screenshot below again.

Repro

I tried making a simple repro but couldn't.

Configuration

.NET Runtime: 8.0.8
CPU: AMD Ryzen 9 5900X 12-Core Processor 3.70 GHz
RAM: 32 GB
Windows info:

Edition	Windows 11 Pro
Version	23H2
Installed on	‎17/‎12/‎2023
OS build	22631.4112
Experience	Windows Feature Experience Pack 1000.22700.1034.0
Regression?

Not that I am aware of.

Data

Integration test backtrace view of the critical section contentions in the JIT, almost all of it is the reportInliningInfo:

image

"Normal" program run backtrace:

image

Dominant language
C#
Stars
18.3k
Forks
5.6k
PR metrics pending

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from dotnet/runtime

All issues in dotnet/runtime

Similar issues

More C# issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.