Skip to content

Move Struct marshaling to transient IL and adjust LayoutClass marshaling IL stub handling#125352

Open
jkoritzinsky wants to merge 25 commits intodotnet:mainfrom
jkoritzinsky:structmarshal-transient-il
Open

Move Struct marshaling to transient IL and adjust LayoutClass marshaling IL stub handling#125352
jkoritzinsky wants to merge 25 commits intodotnet:mainfrom
jkoritzinsky:structmarshal-transient-il

Conversation

@jkoritzinsky
Copy link
Member

Today, we emit IL for struct/layout class marshaling similar to how we handle P/Invoke, Delegate, and COM interop marshalling stubs: We emit an IL stub always, put it in the IL stub cache, etc.

However, struct marshalling stubs have some weird aspects to them today:

  1. We use one stub for all three operations: marshal managed->unmanaged, marshal unmanaged->managed, cleanup. We handle this today with a manual if-else decision tree (because our runtime IL emit tooling doesn't support switch) to jump to the right block.
  2. Struct marshaling IL stubs are the only IL stubs that can be called directly in IL (from other struct marshaling IL stubs). As a result, we need to have more complicated handling for the IL stub's compile time state, just for struct marshaling stubs.
  3. Struct marshaling stubs don't use a significant portion of the logic used with the rest of the marshaling stubs, so we've had to basically no-op those paths for struct marshaling.
  4. The IL stub cache is too slow for common lookups from APIs like Marshal.StructureToPtr, so we had to add an additional faster caching layer in EEMarshalingData.

This PR moves struct marshaling to be handled via Transient IL, with the marshaling logic presented as the IL for each instantiation of System.StubHelpers.StructureMarshaler<T>.

I wanted to do the same for layout-class marshaling, but it's not possible to do so because there's no way to opt-out of generic sharing. However, I was able to structure it similarly by having layout class marshalling continue to generate IL stubs. In the layout-class case, the stubs are no longer cached alongside the rest of the interop stubs with all of that machinery. Layout-class stubs are cached on the managed side in statics in System.StubHelpers.LayoutClassMarshaler<T>, with a lock to ensure we don't double-generate the stubs.

This allows us to remove a decent amount of logic around struct marshaling IL stubs and how they interact with P/Invoke stubs. Additionally, it paves a path towards moving P/Invoke IL bodies to use transient IL bodies instead of IL stubs for non-shared cases.

@dotnet-policy-service
Copy link
Contributor

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors CoreCLR struct and layout-class marshalling stub generation by moving struct marshalling to transient IL (per System.StubHelpers.StructureMarshaler<T> instantiation) and simplifying layout-class marshalling by generating dedicated IL stubs cached in managed statics (System.StubHelpers.LayoutClassMarshaler<T>), reducing reliance on the general IL stub cache and related runtime machinery.

Changes:

  • Generate struct marshalling bodies via transient IL (MethodDesc::TryGenerateTransientILImplementation) instead of cached IL stubs.
  • Add a new QCall to create layout-class marshal stubs on demand and cache them in managed code.
  • Rework field-marshalling flag handling and remove struct-stub-specific caching/compile-time-state paths.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/coreclr/vm/stubhelpers.h Adds QCall declaration for creating layout-class marshal stubs.
src/coreclr/vm/stubhelpers.cpp Implements StubHelpers_CreateLayoutClassMarshalStub QCall.
src/coreclr/vm/stubgen.h Adjusts struct marshal stub argument indexing; adds transient IL generation hook.
src/coreclr/vm/qcallentrypoints.cpp Removes old struct marshal stub QCall entry; adds new layout-class stub QCall entry.
src/coreclr/vm/prestub.cpp Wires struct marshalling transient IL generation into prestub transient IL path.
src/coreclr/vm/olevariant.h Removes managed-marshaler-code parameters from SAFEARRAY/record marshaling helpers.
src/coreclr/vm/olevariant.cpp Updates SAFEARRAY/record marshaling paths to use new struct/layout marshalling entrypoints.
src/coreclr/vm/mlinfo.h Reassigns marshal flag bits and adds new flags for struct marshalling scenarios; updates GenerateFieldIL signature.
src/coreclr/vm/mlinfo.cpp Updates field IL generation to respect marshal flags and avoid invalid trailing NOPs.
src/coreclr/vm/metasig.h Updates metasig definitions for updated marshaler entrypoints.
src/coreclr/vm/marshalnative.h Replaces struct-stub QCall with MarshalNative_HasLayout QCall declaration.
src/coreclr/vm/marshalnative.cpp Implements MarshalNative_HasLayout QCall.
src/coreclr/vm/ilstubresolver.h Removes struct-marshal-specific loader-heap compile-time-state support.
src/coreclr/vm/ilstubresolver.cpp Simplifies generated IL allocation and compile-time-state freeing logic.
src/coreclr/vm/ilstubcache.cpp Removes struct marshal stub special-casing for keeping compile-time state alive.
src/coreclr/vm/ilmarshalers.h Adds new marshal-flag helpers and updates field-marshalling emission paths.
src/coreclr/vm/ilmarshalers.cpp Switches struct/layout marshalling call sites to new method-based entrypoints; updates array marshaller creation.
src/coreclr/vm/dllimport.h Introduces MarshalOperation enum; replaces struct marshal stub APIs with new helpers.
src/coreclr/vm/dllimport.cpp Implements transient IL generation for struct marshaling; adds layout-class stub generation; adds GetStructMarshallingMethod.
src/coreclr/vm/dispparammarshaler.cpp Updates record/array marshaling to use new struct/layout marshalling method entrypoints.
src/coreclr/vm/dispatchinfo.cpp Removes struct-stub lookup for SAFEARRAY element record marshaling.
src/coreclr/vm/corelib.h Adds binder entries for StructureMarshaler<T> and LayoutClassMarshaler<T> methods; updates marshaler metasigs.
src/coreclr/tools/aot/ILCompiler.ReadyToRun/Compiler/ReadyToRunXmlRootProvider.cs Adds null-check when instantiation isn’t possible for generic roots.
src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs Adds StructureMarshaler<T> and LayoutClassMarshaler<T> managed helpers and managed caching for layout-class stubs.
src/coreclr/System.Private.CoreLib/src/System/Runtime/InteropServices/Marshal.CoreCLR.cs Replaces struct-stub-based marshalling with layout probing + cached per-type marshal method entrypoints.
src/coreclr/System.Private.CoreLib/src/System/Runtime/CompilerServices/RuntimeHelpers.CoreCLR.cs Minor formatting-only change (blank line).

@AaronRobinsonMSFT
Copy link
Member

Would it be possible to do this change in two PRs? There are a lot of concerns here to try and reconcile in my head in a single PR.

@jkoritzinsky
Copy link
Member Author

Would it be possible to do this change in two PRs? There are a lot of concerns here to try and reconcile in my head in a single PR.

How would you like it split?

I can try splitting it, but I'm not sure what split would end up making the code any easier to review.

…e for boxed struct inputs (allows simplifying the use cases that have `object` inputs.
Copilot AI review requested due to automatic review settings March 10, 2026 18:32
Copilot AI review requested due to automatic review settings March 13, 2026 00:53
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.

Copilot AI review requested due to automatic review settings March 13, 2026 01:27
@jkoritzinsky jkoritzinsky force-pushed the structmarshal-transient-il branch from de8b997 to ace58eb Compare March 13, 2026 01:27
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.

…for the error logic as well (will throw during IL generation)
Copilot AI review requested due to automatic review settings March 14, 2026 04:28
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.

Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Copilot AI review requested due to automatic review settings March 16, 2026 20:06
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 1 comment.

…Inlining wrapper to ensure exception is handled when expected
Copilot AI review requested due to automatic review settings March 19, 2026 20:03
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

4 participants