Publish reuses stale ReadyToRun output after RuntimeFrameworkVersion changes, so the app ships precompiled code the bundled runtime refuses

[jackdemori]

Summary

If I publish a project twice from the same tree and the only thing I change is the runtime version it bundles, the second publish resolves the correct crossgen2 for the new runtime and then never runs it. The ReadyToRun images from the first publish are copied through unchanged.

When the two runtimes are different major versions this is silently fatal to performance. ReadyToRun images are only read by the runtime major version that produced them (MINIMUM_READYTORUN_MAJOR_VERSION equals READYTORUN_MAJOR_VERSION in every release branch), so the bundled runtime refuses every image and just-in-time compiles the whole application. The publish succeeds, the app is correct, and no warning is printed anywhere.

The same command from a clean tree produces the right output. So whether a build is fast or slow depends on what happens to be sitting in obj.

Steps to reproduce

A console project targeting net10.0 with PublishReadyToRun and RuntimeIdentifier set. The program prints the ReadyToRun format major version out of its own PE header and out of the core library, then asks the runtime whether its own code was precompiled: it reads System.Runtime.JitInfo.GetCompiledMethodCount(), calls one MethodImplOptions.NoInlining method of its own, and reads the count again. A count that moves means the runtime compiled a method that was supposed to be precompiled. The full sources are at the bottom of this issue.

$ ./gen-workload.sh 2000 Workload.cs
$ rm -rf obj bin

$ dotnet publish -c Release -o pubX -p:SelfContained=true -v:detailed | grep "Added Crossgen2"
  Added Crossgen2 runtime pack 'Microsoft.NETCore.App.Crossgen2.osx-arm64@10.0.10'

$ ./pubX/R2RProbe
runtime           .NET 10.0.8
app image format  RTR 16.0
core image format RTR 16.0
probe precompiled yes
workload jitted   0
workload ms       1

$ dotnet publish -c Release -o pubY -p:SelfContained=true -p:RuntimeFrameworkVersion=11.0.0-preview.7.26381.103 -v:detailed | grep "Added Crossgen2"
  Added Crossgen2 runtime pack 'Microsoft.NETCore.App.Crossgen2.osx-arm64@11.0.0-preview.7.26381.103'

$ ./pubY/R2RProbe
runtime           .NET 11.0.0-preview.7.26381.103
app image format  RTR 16.0
core image format RTR 25.0
probe precompiled no
workload jitted   2001
workload ms       45

The second publish resolves crossgen2 11.0.0-preview.7, downloads it, and ships an image the .NET 10 crossgen2 produced. The intermediate image at obj/Release/net10.0/osx-arm64/R2R/R2RProbe.dll still carries the timestamp and the bytes of the first publish, so the compilation step was skipped, not repeated with a different compiler. The two published app images are byte-identical:

$ md5 -q pubX/R2RProbe.dll pubY/R2RProbe.dll
8695a28f12c7c59c64aca8e7976ae478
8695a28f12c7c59c64aca8e7976ae478

Delete obj and run the second command on its own and it is correct: the app image is RTR 25.0, the probe reports precompiled, and nothing is compiled at runtime.

Why I care about it

I bundle a runtime with a large language server, about 200 MB of ReadyToRun images across 181 files. I moved the bundled runtime forward by one major version to measure a file watcher fix, using nothing but RuntimeFrameworkVersion, and my published server lost every byte of its precompiled code. It took me a while to find, because everything still worked.

On a 96 project solution, loaded cold, the analysis phase went from 7356 ms to 8875 ms at the medians of five interleaved runs. On a small project it went from 1330 ms to 1985 ms, a 49 percent rise. Rebuilding from a clean tree, changing nothing else, gave back 1154 ms of the 1519 ms. I chased garbage collection first, then thread pool behaviour, then the just-in-time compiler, before I thought to read the ReadyToRun header out of the published bytes. A one line warning would have saved all of it.

What I would expect

Two things, and the first matters more.

1. The ReadyToRun compilation step should treat the crossgen2 pack identity as an input to its up-to-date check, so changing the target runtime recompiles rather than reuses.
2. Publishing should warn when the ReadyToRun format it produced cannot be read by the runtime it bundles. That check is cheap: the major version is at a fixed offset in the managed native header, and the SDK already knows both versions.

Environment

• macOS 27.0 (Darwin 27.0.0), Apple M1 Pro, arm64
• SDK 11.0.100-preview.7.26381.103, also reproduced with the 10.0.x line
• Target framework net10.0, runtime identifier osx-arm64, PublishReadyToRun=true, SelfContained=true

Repro sources

Workload.cs is generated by the script, 2000 small NoInlining methods, so the JIT cost is visible without depending on any real codebase. Change the RuntimeIdentifier for another platform.

R2RProbe.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <RuntimeIdentifier>osx-arm64</RuntimeIdentifier>
    <SelfContained>false</SelfContained>
    <PublishReadyToRun>true</PublishReadyToRun>
    <AssemblyName>R2RProbe</AssemblyName>
    <RootNamespace>R2RProbe</RootNamespace>
    <InvariantGlobalization>true</InvariantGlobalization>
  </PropertyGroup>
</Project>

Program.cs

using System.Diagnostics;
using System.Reflection;
using System.Runtime;
using System.Runtime.CompilerServices;
using System.Runtime.InteropServices;

internal static class Program
{
    private static int Main()
    {
        Console.WriteLine($"runtime           {RuntimeInformation.FrameworkDescription}");
        Console.WriteLine($"app image format  {ReadyToRunFormat(Assembly.GetExecutingAssembly().Location)}");
        Console.WriteLine($"core image format {ReadyToRunFormat(typeof(object).Assembly.Location)}");

        long before = JitInfo.GetCompiledMethodCount();
        Probe(1);
        long after = JitInfo.GetCompiledMethodCount();
        Console.WriteLine($"probe precompiled {(after == before ? "yes" : "no")}");

        long methodsBefore = JitInfo.GetCompiledMethodCount();
        TimeSpan jitBefore = JitInfo.GetCompilationTime();
        Stopwatch clock = Stopwatch.StartNew();
        long sum = Workload.Run();
        clock.Stop();
        Console.WriteLine($"workload ms       {clock.ElapsedMilliseconds}");
        Console.WriteLine($"workload jitted   {JitInfo.GetCompiledMethodCount() - methodsBefore}");
        Console.WriteLine($"workload jit ms   {(long)(JitInfo.GetCompilationTime() - jitBefore).TotalMilliseconds}");
        Console.WriteLine($"process jitted    {JitInfo.GetCompiledMethodCount()}");
        Console.WriteLine($"process jit ms    {(long)JitInfo.GetCompilationTime().TotalMilliseconds}");
        Console.WriteLine($"checksum          {sum}");
        return 0;
    }

    [MethodImpl(MethodImplOptions.NoInlining)]
    private static int Probe(int value) => value + 1;

    private static string ReadyToRunFormat(string path)
    {
        if (string.IsNullOrEmpty(path) || !File.Exists(path))
        {
            return "unknown";
        }

        byte[] image = File.ReadAllBytes(path);
        int peOffset = BitConverter.ToInt32(image, 0x3c);
        int coff = peOffset + 4;
        int optionalSize = BitConverter.ToUInt16(image, coff + 16);
        int optional = coff + 20;
        int directories = optional + (BitConverter.ToUInt16(image, optional) == 0x10b ? 0x60 : 0x70);
        int corRva = BitConverter.ToInt32(image, directories + (14 * 8));
        if (corRva == 0)
        {
            return "not managed";
        }

        int sections = BitConverter.ToUInt16(image, coff + 2);
        int sectionTable = optional + optionalSize;
        int corOffset = ToFileOffset(image, sectionTable, sections, corRva);
        int nativeRva = BitConverter.ToInt32(image, corOffset + 64);
        if (nativeRva == 0)
        {
            return "il only";
        }

        int nativeOffset = ToFileOffset(image, sectionTable, sections, nativeRva);
        string signature = System.Text.Encoding.ASCII.GetString(image, nativeOffset, 3);
        ushort major = BitConverter.ToUInt16(image, nativeOffset + 4);
        ushort minor = BitConverter.ToUInt16(image, nativeOffset + 6);
        return $"{signature} {major}.{minor}";
    }

    private static int ToFileOffset(byte[] image, int sectionTable, int sections, int rva)
    {
        for (int i = 0; i < sections; i++)
        {
            int header = sectionTable + (i * 40);
            int virtualSize = BitConverter.ToInt32(image, header + 8);
            int virtualAddress = BitConverter.ToInt32(image, header + 12);
            int rawOffset = BitConverter.ToInt32(image, header + 20);
            if (rva >= virtualAddress && rva < virtualAddress + Math.Max(virtualSize, 1))
            {
                return rawOffset + (rva - virtualAddress);
            }
        }

        return 0;
    }
}

gen-workload.sh

#!/bin/bash
set -eu
count=${1:-2000}
out=${2:-Workload.cs}
{
  echo "using System.Runtime.CompilerServices;"
  echo
  echo "internal static class Workload"
  echo "{"
  echo "    internal static long Run()"
  echo "    {"
  echo "        long total = 0;"
  i=0
  while [ $i -lt $count ]; do
    echo "        total += Step$i(total);"
    i=$((i+1))
  done
  echo "        return total;"
  echo "    }"
  echo
  i=0
  while [ $i -lt $count ]; do
    echo "    [MethodImpl(MethodImplOptions.NoInlining)]"
    echo "    private static long Step$i(long seed)"
    echo "    {"
    echo "        long value = seed ^ $i;"
    echo "        for (int i = 0; i < 64; i++)"
    echo "        {"
    echo "            value = (value * 31) + (i ^ $i);"
    echo "        }"
    echo "        return value & 0xffff;"
    echo "    }"
    i=$((i+1))
  done
  echo "}"
} > "$out"
echo "wrote $out with $count methods"

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• ReadyToRun property changes do not invalidate incremental build outputs #55436

[jackdemori]

Related, but not a duplicate, I believe. #55436 is _CreateR2RImages reusing outputs when the effective ReadyToRun configuration (composite mode) changes between publishes. This issue is the same target reusing outputs when the compiler itself changes: RuntimeFrameworkVersion resolves a different crossgen2 pack, which is downloaded and then never run. The underlying gap is shared: the target's incremental inputs track the IL assemblies but not the load-bearing properties or the resolved crossgen2 identity. The property-tracking pattern baronfel describes in #55436 would cover both cases, provided the crossgen2 pack identity joins the tracked set.

One part of this report is not covered there: the ask for a publish-time warning when the ReadyToRun format produced cannot be read by the runtime being bundled. That check would have surfaced both issues immediately instead of shipping silently slow output.

Happy to have this transferred to dotnet/sdk if that is the better home.