Run trimming tests as AOT tests by MichalStrehovsky · Pull Request #101229 · dotnet/runtime

MichalStrehovsky · 2024-04-18T08:57:23Z

Not everything is passing, so I baselined this. Some we'll probably exclude permanently, others are more concerning and we need to determine if it's test issues or product issues.

Cc @dotnet/ilc-contrib

Not everything is passing, so I baselined this. Some we'll probably exclude permanently, others are more concerning and we need to determine if it's test issues or product issues.

dotnet-policy-service · 2024-04-18T08:57:49Z

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

MichalStrehovsky · 2024-04-18T09:01:16Z

/azp run runtime-nativeaot-outerloop

azure-pipelines · 2024-04-18T09:01:24Z

Azure Pipelines successfully started running 1 pipeline(s).

eerhardt · 2024-04-18T15:20:31Z

src/libraries/tests.proj

+    <DisabledNativeAotTestAppProjects Include="$(MSBuildThisFileDirectory)\System.Runtime\tests\System.Runtime.Tests\TrimmingTests\System.Runtime.TrimmingTests.proj" />
+    <DisabledNativeAotTestAppProjects Include="$(MSBuildThisFileDirectory)\System.Text.Json\tests\System.Text.Json.Tests\TrimmingTests\System.Text.Json.TrimmingTests.proj" />
+
+    <NativeAotTestAppProjects Include="$(MSBuildThisFileDirectory)*\tests\**\*.NativeAotTests.proj;$(MSBuildThisFileDirectory)*\tests\**\*.TrimmingTests.proj"


Should we just get rid of the NativeAotTests.proj extension (there is only 1 currently in the repo) and instead just have TrimmingTests.proj, which are run as both PublishTrimmed and PublishAot?

If you don't have concerns about running the NativeAotTests with trimming as well, that sounds good to me too.

Trimming is generally a subset of AOT.

This is the only NativeAotTests today:

runtime/src/libraries/System.Diagnostics.DiagnosticSource/tests/NativeAotTests/DiagnosticSourceEventSourceTests.cs

Lines 45 to 84 in 4760cf0

public static int Main()

{

DiagnosticSource diagnosticSource = new DiagnosticListener("TestDiagnosticListener");

using (var listener = new TestEventListener())

{

var data = new EventData()

{

Id = Guid.NewGuid(),

};

Write(diagnosticSource, "Test.Start", data);

if (!(listener.LogDataPayload?.Count == 3 &&

(string)listener.LogDataPayload[0] == "TestDiagnosticListener" &&

(string)listener.LogDataPayload[1] == "Test.Start"))

{

return -1;

}

object[] args = (object[])listener.LogDataPayload[2];

if (args.Length != 2)

{

return -2;

}

IDictionary<string, object> arg = (IDictionary<string, object>)args[0];

if (!((string)arg["Key"] == "Id" && (string)arg["Value"] == data.Id.ToString()))

{

return -3;

}

arg = (IDictionary<string, object>)args[1];

if (!((string)arg["Key"] == "*Enumerate" && (string)arg["Value"] == "1,2,3"))

{

return -4;

}

return 100;

}

}

It should work just fine with PublishTrimmed. It was written to test the MakeGenericType deep inside of DiagnosticSourceEventSource works with NativeAOT.

I think it would be best to just have 1 "kind of test" and it runs for both PublishTrimmed and PublishAot. It will make it easier to write these, and it will add a bit of coverage to both scenarios.

eerhardt · 2024-04-18T15:20:54Z

eng/testing/linker/trimmingTests.targets

    <MSBuild Projects="@(TestConsoleApps)"
             Targets="Publish"
-             Properties="Configuration=$(Configuration);BuildProjectReferences=false;TargetOS=$(TargetOS);TargetArchitecture=$(TargetArchitecture)" />
+             Properties="Configuration=$(Configuration);BuildProjectReferences=false;TargetOS=$(TargetOS);TargetArchitecture=$(TargetArchitecture);_IsPublishing=true" />


Why is this change necessary?

The tests weren't building because they couldn't find AppHost. This is running the Publish target directly, which is very problematic because it bypasses logic in the SDK. We have been having a conversation with the SDK team for months. The fix is to do what real publish does and that's setting _IsPublishing. We do that in multiple places in this repo even though it's used/defined by the SDK.

This reverts commit 156a359.

eerhardt

LGTM. Thank you for making this happen.

eerhardt · 2024-04-19T13:24:13Z

eng/testing/linker/trimmingTests.targets

      <_additionalPropertiesString>@(_propertiesAsItems->'&lt;%(Identity)&gt;%(Value)&lt;/%(Identity)&gt;', '%0a    ')</_additionalPropertiesString>
    </PropertyGroup>

+    <!-- RunNativeAotTestApps trumps TestTrimming, same as PublishAot trumps PublishTrimmed -->


(nit) this comment doesn't align with the code. The comment says one trumps the other. The code says if they are both true it is an error.

MichalStrehovsky · 2024-04-22T09:43:55Z

/ba-g slow mac

Not everything is passing, so I baselined this. Some we'll probably exclude permanently, others are more concerning and we need to determine if it's test issues or product issues.

Run trimming tests as AOT tests

8adc238

Not everything is passing, so I baselined this. Some we'll probably exclude permanently, others are more concerning and we need to determine if it's test issues or product issues.

ghost added the area-Infrastructure-libraries label Apr 18, 2024

dotnet-policy-service bot assigned MichalStrehovsky Apr 18, 2024

MichalStrehovsky mentioned this pull request Apr 18, 2024

Go over trimming tests we disabled for AOT #101228

Closed

MichalStrehovsky requested a review from eerhardt April 18, 2024 08:59

MichalStrehovsky mentioned this pull request Apr 18, 2024

Run DependencyInjection trimming tests as native AOT. #101129

Closed

This was referenced Apr 18, 2024

CI error: System.Net.Quic.QuicException: The connection timed out from inactivity #91757

Closed

[wasm] System.Text.Json.Tests running out of memory #98578

Closed

Timeout in System.Text.Json.Tests.Utf8JsonWriterTests.WriteNumbers #101193

Closed

Dummy change to trigger the legs that test this

156a359

eerhardt reviewed Apr 18, 2024

View reviewed changes

MichalStrehovsky added 3 commits April 19, 2024 09:38

Revert "Dummy change to trigger the legs that test this"

83799e8

This reverts commit 156a359.

FB

4150e0b

Different pipeline

b5e356c

build-analysis bot mentioned this pull request Apr 19, 2024

[wasm] Assertion: should not be reached at D:/a/_work/1/s/src/mono/mono/sgen/sgen-scan-object.h:93 in System.Reflection.Tests #95805

Closed

MichalStrehovsky mentioned this pull request Apr 19, 2024

System.Net.Http.Functional.Tests.HttpMetricsTest_Http20.AllSocketsHttpHandlerCounters_Success_Recorded item comparison failure #101285

Closed

eerhardt approved these changes Apr 19, 2024

View reviewed changes

FB

b070d3d

MichalStrehovsky merged commit d8eecb7 into dotnet:main Apr 22, 2024

MichalStrehovsky deleted the aottests branch April 22, 2024 09:44

build-analysis bot mentioned this pull request Apr 22, 2024

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." dotnet/dnceng#1883

Open

3 tasks

github-actions bot locked and limited conversation to collaborators May 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run trimming tests as AOT tests#101229

Run trimming tests as AOT tests#101229
MichalStrehovsky merged 6 commits intodotnet:mainfrom
MichalStrehovsky:aottests

MichalStrehovsky commented Apr 18, 2024 •

edited

Loading

Uh oh!

dotnet-policy-service bot commented Apr 18, 2024

Uh oh!

MichalStrehovsky commented Apr 18, 2024

Uh oh!

azure-pipelines bot commented Apr 18, 2024

Uh oh!

eerhardt Apr 18, 2024

Uh oh!

MichalStrehovsky Apr 18, 2024 •

edited

Loading

Uh oh!

eerhardt Apr 18, 2024

Uh oh!

eerhardt Apr 18, 2024

Uh oh!

MichalStrehovsky Apr 18, 2024

Uh oh!

eerhardt left a comment

Uh oh!

eerhardt Apr 19, 2024

Uh oh!

MichalStrehovsky commented Apr 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	public static int Main()
	{
	DiagnosticSource diagnosticSource = new DiagnosticListener("TestDiagnosticListener");
	using (var listener = new TestEventListener())
	{
	var data = new EventData()
	{
	Id = Guid.NewGuid(),
	};

	Write(diagnosticSource, "Test.Start", data);

	if (!(listener.LogDataPayload?.Count == 3 &&
	(string)listener.LogDataPayload[0] == "TestDiagnosticListener" &&
	(string)listener.LogDataPayload[1] == "Test.Start"))
	{
	return -1;
	}

	object[] args = (object[])listener.LogDataPayload[2];
	if (args.Length != 2)
	{
	return -2;
	}

	IDictionary<string, object> arg = (IDictionary<string, object>)args[0];
	if (!((string)arg["Key"] == "Id" && (string)arg["Value"] == data.Id.ToString()))
	{
	return -3;
	}

	arg = (IDictionary<string, object>)args[1];
	if (!((string)arg["Key"] == "*Enumerate" && (string)arg["Value"] == "1,2,3"))
	{
	return -4;
	}

	return 100;
	}
	}

Conversation

MichalStrehovsky commented Apr 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dotnet-policy-service bot commented Apr 18, 2024

Uh oh!

MichalStrehovsky commented Apr 18, 2024

Uh oh!

azure-pipelines bot commented Apr 18, 2024

Uh oh!

eerhardt Apr 18, 2024

Choose a reason for hiding this comment

Uh oh!

MichalStrehovsky Apr 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eerhardt Apr 18, 2024

Choose a reason for hiding this comment

Uh oh!

eerhardt Apr 18, 2024

Choose a reason for hiding this comment

Uh oh!

MichalStrehovsky Apr 18, 2024

Choose a reason for hiding this comment

Uh oh!

eerhardt left a comment

Choose a reason for hiding this comment

Uh oh!

eerhardt Apr 19, 2024

Choose a reason for hiding this comment

Uh oh!

MichalStrehovsky commented Apr 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MichalStrehovsky commented Apr 18, 2024 •

edited

Loading

MichalStrehovsky Apr 18, 2024 •

edited

Loading