Add PSJsonSerializerV2 for ConvertTo-Json using TruncatingConverterFactory#26654
Add PSJsonSerializerV2 for ConvertTo-Json using TruncatingConverterFactory#26654yotsuda wants to merge 29 commits intoPowerShell:masterfrom
Conversation
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Show resolved
Hide resolved
…Cmdlet/ConvertToJsonCommandV2.cs Co-authored-by: Ilya <[email protected]>
…Cmdlet/ConvertToJsonCommandV2.cs Co-authored-by: Ilya <[email protected]>
74a83c8 to
783f3b3
Compare
|
@yotsuda It seems you lost last commits - I don't see updates. |
|
@iSazonov Sorry, I missed the same pattern in
|
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Show resolved
Hide resolved
|
|
||
| // Convert to string when depth exceeded | ||
| string stringValue = LanguagePrimitives.ConvertTo<string>(pso.ImmediateBaseObjectIsEmpty ? pso : obj); | ||
| writer.WriteStringValue(stringValue); |
There was a problem hiding this comment.
It seems you don't understand my request. I mean if max depth is reached, V1 convert base object to string and process extended/adapted properties too. In V2 we only convert base object to string but don't process extended/adapted properties.
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
…e write-only check to WriteProperty
|
Please see failed tests. $jstr
{
"Major": 2,
"Minor": 3,
"Build": 4,
"Revision": 14,
"MajorRevision": 0,
"MinorRevision": 14,
"Note": "a version object",
"Rev": 14,
"IsOld": true
}What is $jstr in V2? As for |
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
|
|
||
| // GetTypeInfo() has internal caching, no need for additional cache | ||
| var typeInfo = JsonSerializerOptions.Default.GetTypeInfo(type); | ||
| return typeInfo.Kind == JsonTypeInfoKind.None; |
There was a problem hiding this comment.
How do we process type(object) in V1 and V2?
See https://source.dot.net/#System.Text.Json/System/Text/Json/Serialization/Metadata/JsonTypeInfo.cs,1337
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Show resolved
Hide resolved
| public static bool ShouldSkipProperty(PSPropertyInfo prop) | ||
| { | ||
| // Check for Hidden attribute | ||
| if (prop.IsHidden) |
There was a problem hiding this comment.
Interesting, how this (doesn't) works for script-defined classes.
Both issues are fixed in the latest commits. AddMember on JsonObject (Version + ETS): V2 now produces the same output as V1 for Version objects with ETS properties: The fix adds V1-compatible logic: "true primitive types" (string, DateTime, etc.) always serialize as scalars ignoring ETS, while non-primitive STJ scalar types (Version, IPAddress, etc.) serialize as objects when they have ETS properties. [char]127 serialization: Implemented as you suggested - the test now selects expected result based on experimental feature status:
Commit: 8efb723 |
…Cmdlet/ConvertToJsonCommandV2.cs Co-authored-by: Ilya <[email protected]>
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Outdated
Show resolved
Hide resolved
| private void SerializeAsObject(Utf8JsonWriter writer, PSObject pso, JsonSerializerOptions options) | ||
| { | ||
| writer.WriteStartObject(); | ||
| AppendPSProperties(writer, pso, options, PSMemberViewTypes.Extended | PSMemberViewTypes.Adapted); |
There was a problem hiding this comment.
I see V1 does:
- Enumerate properties using reflection.
- Add extended and adapted properties (excluding duplicates from 1.).
It seems if we request adapted properties we get reflection properties too. I guess there may be differences, but I do not know how it works in depth.
@SeeminglyScience @daxian-dbw Could you please help?
There was a problem hiding this comment.
I created Version type object, then add ETS noteproperty with Add-Member, then request properties with PSMemberInfoIntegratingCollection:
- for
BaseI see only the note property, - for
ExtendedI see only the note property, - for
AdaptedI see the note property and native properties.
My understanding is that is how PSObject wrapping works. PSObject is a property bag. So for Base just the ps object is considered. For Extend - only ETS properties as exclusively separate. For Adapted - adapters are involved including .Net Adapter which return native .Net members. Some adapters can be invovled (including XML, CIM/WMI,COM). Property priority is Extended -> Adapded -> Base.
V1 code add native .Net properties by reflection then enumerate Extended and Adapted ones with PSMemberInfoIntegratingCollection and add them only if the name is not already in list. So reflection properties win.
If we use only PSMemberInfoIntegratingCollection in V2 - Extended properties win. It is breaking change.
Perhaps it would be worthwhile to adhere to this native PowerShell behavior, but since historically this cmdlet works differently, it is not difficult for us to do the same by taking properties from the Json Default cache or by psObj.Properties.Match("*", PSMemberTypes.Property); (although it is more expensive).
…ack in WriteProperty
src/Microsoft.PowerShell.Commands.Utility/commands/utility/WebCmdlet/ConvertToJsonCommandV2.cs
Show resolved
Hide resolved
.../powershell/Modules/Microsoft.PowerShell.Utility/ConvertTo-Json.PSJsonSerializerV2.Tests.ps1
Outdated
Show resolved
Hide resolved
|
@yotsuda Please look failed test. |
|
@iSazonov Thank you for pointing this out. I apologize for the delayed response to the CI test failures. I was running CI Test Failures
Root Cause AnalysisThe root cause of the
STJ returns Background: Option C ImplementationIn PR #26637, you suggested Option C to track raw vs PSObject instead of using a hardcoded type list. Based on this, I replaced the hardcoded However, STJ's Two Possible FixesFix#1: STJ-based with Exception List if (type == typeof(Version) || type == typeof(Type))
return false; // Treat as object, not scalar
return typeInfo.Kind == JsonTypeInfoKind.None;
Fix#2: Revert to V1's Explicit Type List return obj is string or char or bool or DateTime or DateTimeOffset
or Guid or Uri or double or float or decimal or BigInteger;
Local Test Results (Fix#1)I tested Fix#1 locally:
Proposed Next Steps
Which fix do you prefer, or do you have a different solution in mind? |
|
@yotsuda Thanks! We need to strive for backward compatibility to make it easier to migrate to an updated cmdlet. The same tests should work for both versions. |
|
@iSazonov Thank you for the feedback! I've applied Fix#1 (Version exception) and fixed the cross-platform test issues in commit 6984cf3. However, I'd like to clarify the difference between Current ImplementationThe code has two separate exclusion mechanisms for different purposes:
V1 Behavior Verification# string/DateTime: ETS properties are IGNORED
$str = "hello"
$str | Add-Member -NotePropertyName ETS -NotePropertyValue "test" -PassThru | ConvertTo-Json
# Output: "hello" (no ETS)
# Version: ETS properties are INCLUDED
$ver = [Version]"1.2.3"
$ver | Add-Member -NotePropertyName ETS -NotePropertyValue "test" -PassThru | ConvertTo-Json -Compress
# Output: {"Major":1,"Minor":2,"Build":3,"Revision":-1,...,"ETS":"test"}Why Not Combine Into One List?If we added The current fix correctly handles Version by:
This matches V1 behavior exactly. Changes in This Commit (6984cf3)
Question: V2-Specific TestsI've created If your goal is to make V2 fully compatible with V1, I should:
Would you like me to pursue full V1 compatibility, or is it acceptable for V2 to have documented behavioral improvements while maintaining compatibility for common use cases? |
|
@yotsuda Thanks for clarify! It was my wrong understanding of Version behavior. I feel there is another issue. If a type has a custom converter, it return JsonTypeInfoKind.None then we consider it as scalar type and call the type converter without out custom depth control. Yes?
I am ok with "non-string dictionary keys support" added. |
|
@iSazonov Thank you for the clarification! Custom Converter + Kind=None
Yes, that's the current behavior. For types with
My understanding is that this is acceptable because these types are "leaf" values:
Note: BigInteger has If you see any issues with this approach, please let me know. Other Breaking ChangesBesides the non-string dictionary keys support, I'm not currently aware of other breaking changes, but I may have missed something. If you have specific scenarios you'd like me to verify, I'd be happy to test them. |
If an user creates custom class with custom converter the class will not be serialized by our reflection converter with our depth control. |
|
@iSazonov Thank you for the clarification. You're right - custom converters bypass our depth control. I investigated how V1 handles this. V1's ApproachV1 never uses
Since the original object is converted to a dictionary before being passed to Newtonsoft.Json, any Options
Which approach do you prefer? |
|
@yotsuda I think it's time to take stock. I feel my expectations that the new code will be simpler, faster and the functionality extensible have not been fulfilled. By creating workarounds, we have lost all the benefits of STJ. |
|
@iSazonov I really appreciate all your detailed reviews and guidance throughout this PR. I agree that maintaining V1 compatibility while keeping V2 simple, fast, and extensible has proven to be a difficult balance. Here's a summary of the challenges we encountered:
Intentional improvement in V2: Non-string dictionary keys are now supported (V1 threw an error). As you noted, these workarounds have added complexity. I hope this code and analysis can serve as useful material for the .NET Runtime discussion. Please let me know if there's anything you'd like me to revise. |
|
@yotsuda Thanks! I opened dotnet/runtime#123424 |
|
@yotsuda dotnet/runtime#123424 was closed. There are huge limitations in STJ design and implementation, it's incredible that they'll change anything in the near future. We have to close the PR. Thanks for your efforts! |
|
@iSazonov Thanks for the clarification and for filing the upstream issue. I understand STJ migration is not feasible given the design limitations. Closing this PR. I'll continue with #26639 for compatibility tests. The test failures there are fixed and it's ready for review. Please share your thoughts on additional tests or cmdlet improvements when you have time. Thanks for your guidance throughout this effort! |
PR Summary
Implements
TruncatingConverterFactorypattern forConvertTo-JsonV2 as discussed with @iSazonov in PR #26637.This PR is an alternative implementation approach based on the Factory pattern, branched from
feature-convertto-json-isazonov-approach.Fixes #5749
PR Context
Background
In PR #26637, @iSazonov suggested implementing the Factory approach in a new PR to evaluate whether it provides a better architecture than the current implementation.
Thanks to @iSazonov's guidance on the technical considerations, this implementation now works correctly.
Solution
This implementation uses a
TruncatingConverterFactorythat dispatches to type-specific converters:JsonConverterPSObject(Extended/Adapted properties)JsonConverterDictionary<T>(Base properties only)JsonConverterEnumerable<T>JsonConverterComposite<T>(Base properties only)Architecture Benefits
Key Differences from PR #26637
RawObjectWrapperclassTruncatingConverterFactorydispatchWriteValueJsonConverterPSObjectwith_basePropertiesOnlyflagPR Checklist
.h,.cpp,.cs,.ps1and.psm1files have the correct copyright headerPSJsonSerializerV2Changes Made
Modified Files
ConvertToJsonCommandV2.csTruncatingConverterFactory- dispatches to appropriate converters based on typeJsonConverterDictionary<T>- handles IDictionary typesJsonConverterEnumerable<T>- handles IEnumerable typesJsonConverterComposite<T>- handles composite objects with Base properties onlyRawObjectWrapperclass_basePropertiesOnlyfield andfilterToPropertyOnlyparameter fromJsonConverterPSObjectWriteValuehelper toJsonSerializerHelperConvertTo-Json.PSJsonSerializerV2.Tests.ps1Testing
Existing Tests (ConvertTo-Json.Tests.ps1)
V2 Specific Tests (ConvertTo-Json.PSJsonSerializerV2.Tests.ps1)
Related Work
Branch Structure
gitGraph commit id: "master" branch system-text-json commit id: "77d8a1b0c" commit id: "159b76387" branch isazonov-approach commit id: "e7dfff962" commit id: "da0c8cbfb" branch factory-test commit id: "test" type: REVERSE checkout isazonov-approach commit id: "6913cb119" commit id: "2ecc51e05" commit id: "2c549207a" tag: "PR #26637" branch factory-v2 commit id: "af393269b" tag: "HEAD" checkout system-text-json commit id: "07cea1dc5" tag: "PR #26624"system-text-jsonisazonov-approachfactory-testfactory-v2