With much help from @hoyosjs, @carlossanlop, and @krwq I think I have figured this out: the problem is that sometimes the crossgen2 package was restored as net6.0 and sometimes as net7.0. I can't identify from the logs where the net7.0 was coming from, but I think I can pass it down from the package build, which should hopefully ensure that it's always 6.0 at the time it matters.

So, digging in more, I'm more confused about what's going on here. I wonder if there's something I don't understand about MSBuild static graph restore.

@rainersigwald Any chance you can provide any insight here? The problem is that the project.assets.json does not seem to consistently get a RID included. It has net6.0 instead of net6.0/centos.7-x64.

The build I'm having trouble with is at https://dev.azure.com/dnceng/public/_build/results?buildId=1709594&view=logs&j=000ba8e3-34f8-51d2-8d06-9b61f10256d0&t=ca9b6e26-8c16-57f9-673e-2bc0dc655f25

The important part of the log is

Microsoft.NETCore.App.Crossgen2 -> 
    Deleting project.assets.json
    Continuing with RID centos.7-x64
    Determining projects to restore...
    Restored /__w/1/s/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler.Diagnostics/ILCompiler.Diagnostics.csproj (in 270 ms).
    Restored /__w/1/s/artifacts/source-build/self/src/src/coreclr/tools/aot/crossgen2/crossgen2.csproj (in 269 ms).
    Restored /__w/1/s/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler.TypeSystem/ILCompiler.TypeSystem.csproj (in 269 ms).
    Restored /__w/1/s/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler.ReadyToRun/ILCompiler.ReadyToRun.csproj (in 269 ms).
    Restored /__w/1/s/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler.DependencyAnalysisFramework/ILCompiler.DependencyAnalysisFramework.csproj (in 29 ms).
    FileContents: /__w/1/s/artifacts/source-build/self/src/artifacts/obj/coreclr/crossgen2/project.assets.json
    {
    "version": 3,
    "targets": {
    "net6.0": {
    "Microsoft.Build.Tasks.Git/1.2.0-beta-22174-02": {
    "type": "package",
...

The important bits are

    Deleting project.assets.json
    Continuing with RID centos.7-x64

and

   "targets": {
    "net6.0": {

I don't understand why that is not "targets": { "net6.0/centos.7-x64". The code here is basically a straightforward MSBuild invocation of the target project:
https://github.com/dotnet/runtime/pull/67636/files#diff-115c40d5f6cdfb8ebea1490c13581e76b60c3430cbf13f45c270fc5ec383daffR36-R45

The RuntimeIdentifier should be a global property, and I don't understand why it's not factoring into the restore.

also + @dsplaisted in case you have any ideas or can point me in the right direction

Even if a RuntimeIdentifier is specified, the assets file will still have a RID-less "targets" entry in it, and it will be the first one. Later on in the file there should also be a target for "net6.0/centos.7-x64".

   "targets": {
    "net6.0": {

@ViktorHofer Anything else high-priority? I've opened an issue for follow-up work.

@jkoritzinsky Does this new strategy also look OK to you? I don't think much has changed except including the RID in the list of RuntimeIdentifiers in the first restore, so even if the assets.json is cached, it will still have compatible content.

@ViktorHofer Done, anything else?

I do have a lot more questions on why we need to restore that project separately outside of the upfront static graph repo restore but I don't think they should block this PR to going in.

Just wanted to make sure that you understand that a lazy just-in-time restore is the thing that eventually causes a build to fail in the middle. I saw such cases with the libraries linker tests which do something similar. But we can discuss that in your follow-up issue, if you want to :)

Makes sense. First thing in the follow-up can be to remove the restore. It may no longer be necessary.