Publish reuses stale ReadyToRun output after RuntimeFrameworkVersion changes, so the app ships precompiled code the bundled runtime refuses
- Dominant language
- C#
- Stars
- 3.2k
- Forks
- 1.3k
- PR merge metrics
- PR metrics pending
Description
### Summary
If I publish a project twice from the same tree and the only thing I change is the runtime version it bundles, the second publish resolves the correct crossgen2 for the new runtime and then never runs it. The ReadyToRun images from the first publish are copied through unchanged.
When the two runtimes are different major versions this is silently fatal to performance. ReadyToRun images are only read by the runtime major version that produced them (`MINIMUM_READYTORUN_MAJOR_VERSION` equals `READYTORUN_MAJOR_VERSION` in every release branch), so the bundled runtime refuses every image and just-in-time compiles the whole application. The publish succeeds, the app is correct, and no warning is printed anywhere.
The same command from a clean tree produces the right output. So whether a build is fast or slow depends on what happens to be sitting in `obj`.
### Steps to reproduce
A console project targeting `net10.0` with `PublishReadyToRun` and `RuntimeIdentifier` set. The program prints the ReadyToRun format major version out of its own PE header and out of the core library, then asks the runtime whether its own code was precompiled: it reads `System.Runtime.JitInfo.GetCompiledMethodCount()`, calls one `MethodImplOptions.NoInlining` method of its own, and reads the count again. A count that moves means the runtime compiled a method that was supposed to be precompiled. The full sources are at the bottom of this issue.
```
$ ./gen-workload.sh 2000 Workload.cs
$ rm -rf obj bin
$ dotnet publish -c Release -o pubX -p:SelfContained=true -v:detailed | grep "Added Crossgen2"
Added Crossgen2 runtime pack 'Microsoft.NETCore.App.Crossgen2.osx-arm64@10.0.10'
$ ./pubX/R2RProbe
runtime .NET 10.0.8
app image format RTR 16.0
core image format RTR 16.0
probe precompiled yes
workload jitted 0
workload ms 1
$ dotnet publish -c Release -o pubY -p:SelfContained=true -p:RuntimeFrameworkVersion=11.0.0-preview.7.26381.103 -v:detailed | grep "Added Crossgen2"
Added Crossgen2 runtime pack 'Microsoft.NETCore.App.Crossgen2.osx-arm64@11.0.0-preview.7.26381.103'
$ ./pubY/R2RProbe
runtime .NET 11.0.0-preview.7.26381.103
app image format RTR 16.0
core image format RTR 25.0
probe precompiled no
workload jitted 2001
workload ms 45
```
The second publish resolves crossgen2 11.0.0-preview.7, downloads it, and ships an image the .NET 10 crossgen2 produced. The intermediate image at `obj/Release/net10.0/osx-arm64/R2R/R2RProbe.dll` still carries the timestamp and the bytes of the first publish, so the compilation step was skipped, not repeated with a different compiler. The two published app images are byte-identical:
```
$ md5 -q pubX/R2RProbe.dll pubY/R2RProbe.dll
8695a28f12c7c59c64aca8e7976ae478
8695a28f12c7c59c64aca8e7976ae478
```
Delete `obj` and run the second command on its own and it is correct: the app image is RTR 25.0, the probe reports precompiled, and nothing is compiled at runtime.
### Why I care about it
I bundle a runtime with a large language server, about 200 MB of ReadyToRun images across 181 files. I moved the bundled runtime forward by one major version to measure a file watcher fix, using nothing but `RuntimeFrameworkVersion`, and my published server lost every byte of its precompiled code. It took me a while to find, because everything still worked.
On a 96 project solution, loaded cold, the analysis phase went from 7356 ms to 8875 ms at the medians of five interleaved runs. On a small project it went from 1330 ms to 1985 ms, a 49 percent rise. Rebuilding from a clean tree, changing nothing else, gave back 1154 ms of the 1519 ms. I chased garbage collection first, then thread pool behaviour, then the just-in-time compiler, before I thought to read the ReadyToRun header out of the published bytes. A one line warning would have saved all of it.
### What I would expect
Two things, and the first matters more.
1. The ReadyToRun compilation step should treat the crossgen2 pack identity as an input to its up-to-date check, so changing the target runtime recompiles rather than reuses.
2. Publishing should warn when the ReadyToRun format it produced cannot be read by the runtime it bundles. That check is cheap: the major version is at a fixed offset in the managed native header, and the SDK already knows both versions.
### Environment
- macOS 27.0 (Darwin 27.0.0), Apple M1 Pro, arm64
- SDK 11.0.100-preview.7.26381.103, also reproduced with the 10.0.x line
- Target framework `net10.0`, runtime identifier `osx-arm64`, `PublishReadyToRun=true`, `SelfContained=true`
### Repro sources
`Workload.cs` is generated by the script, 2000 small `NoInlining` methods, so the JIT cost is visible without depending on any real codebase. Change the `RuntimeIdentifier` for another platform.
R2RProbe.csproj
```xml
Exe
net10.0
enable
enable
osx-arm64
false
true
R2RProbe
R2RProbe
true
```
Program.cs
```csharp
using System.Diagnostics;
using System.Reflection;
using System.Runtime;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;
internal static class Program
{
private static int Main()
{
Console.WriteLine($"runtime {RuntimeInformation.FrameworkDescription}");
Console.WriteLine($"app image format {ReadyToRunFormat(Assembly.GetExecutingAssembly().Location)}");
Console.WriteLine($"core image format {ReadyToRunFormat(typeof(object).Assembly.Location)}");
long before = JitInfo.GetCompiledMethodCount();
Probe(1);
long after = JitInfo.GetCompiledMethodCount();
Console.WriteLine($"probe precompiled {(after == before ? "yes" : "no")}");
long methodsBefore = JitInfo.GetCompiledMethodCount();
TimeSpan jitBefore = JitInfo.GetCompilationTime();
Stopwatch clock = Stopwatch.StartNew();
long sum = Workload.Run();
clock.Stop();
Console.WriteLine($"workload ms {clock.ElapsedMilliseconds}");
Console.WriteLine($"workload jitted {JitInfo.GetCompiledMethodCount() - methodsBefore}");
Console.WriteLine($"workload jit ms {(long)(JitInfo.GetCompilationTime() - jitBefore).TotalMilliseconds}");
Console.WriteLine($"process jitted {JitInfo.GetCompiledMethodCount()}");
Console.WriteLine($"process jit ms {(long)JitInfo.GetCompilationTime().TotalMilliseconds}");
Console.WriteLine($"checksum {sum}");
return 0;
}
[MethodImpl(MethodImplOptions.NoInlining)]
private static int Probe(int value) => value + 1;
private static string ReadyToRunFormat(string path)
{
if (string.IsNullOrEmpty(path) || !File.Exists(path))
{
return "unknown";
}
byte[] image = File.ReadAllBytes(path);
int peOffset = BitConverter.ToInt32(image, 0x3c);
int coff = peOffset + 4;
int optionalSize = BitConverter.ToUInt16(image, coff + 16);
int optional = coff + 20;
int directories = optional + (BitConverter.ToUInt16(image, optional) == 0x10b ? 0x60 : 0x70);
int corRva = BitConverter.ToInt32(image, directories + (14 * 8));
if (corRva == 0)
{
return "not managed";
}
int sections = BitConverter.ToUInt16(image, coff + 2);
int sectionTable = optional + optionalSize;
int corOffset = ToFileOffset(image, sectionTable, sections, corRva);
int nativeRva = BitConverter.ToInt32(image, corOffset + 64);
if (nativeRva == 0)
{
return "il only";
}
int nativeOffset = ToFileOffset(image, sectionTable, sections, nativeRva);
string signature = System.Text.Encoding.ASCII.GetString(image, nativeOffset, 3);
ushort major = BitConverter.ToUInt16(image, nativeOffset + 4);
ushort minor = BitConverter.ToUInt16(image, nativeOffset + 6);
return $"{signature} {major}.{minor}";
}
private static int ToFileOffset(byte[] image, int sectionTable, int sections, int rva)
{
for (int i = 0; i < sections; i++)
{
int header = sectionTable + (i * 40);
int virtualSize = BitConverter.ToInt32(image, header + 8);
int virtualAddress = BitConverter.ToInt32(image, header + 12);
int rawOffset = BitConverter.ToInt32(image, header + 20);
if (rva >= virtualAddress && rva < virtualAddress + Math.Max(virtualSize, 1))
{
return rawOffset + (rva - virtualAddress);
}
}
return 0;
}
}
```
gen-workload.sh
```bash
#!/bin/bash
set -eu
count=${1:-2000}
out=${2:-Workload.cs}
{
echo "using System.Runtime.CompilerServices;"
echo
echo "internal static class Workload"
echo "{"
echo " internal static long Run()"
echo " {"
echo " long total = 0;"
i=0
while [ $i -lt $count ]; do
echo " total += Step$i(total);"
i=$((i+1))
done
echo " return total;"
echo " }"
echo
i=0
while [ $i -lt $count ]; do
echo " [MethodImpl(MethodImplOptions.NoInlining)]"
echo " private static long Step$i(long seed)"
echo " {"
echo " long value = seed ^ $i;"
echo " for (int i = 0; i < 64; i++)"
echo " {"
echo " value = (value * 31) + (i ^ $i);"
echo " }"
echo " return value & 0xffff;"
echo " }"
i=$((i+1))
done
echo "}"
} > "$out"
echo "wrote $out with $count methods"
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the publish commands and the ReadyToRun intermediate image at obj/Release/net10.0/osx-arm64/R2R/R2RProbe.dll, then trace how the crossgen2 runtime pack identity participates in the up-to-date check. Reproduce the two publishes and use the R2RProbe output and image hashes to verify that changing RuntimeFrameworkVersion recompiles the image and preserves precompiled execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- build-system, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100