Skip to content

Add PSJsonSerializerV2 for ConvertTo-Json using TruncatingConverterFactory#26654

Closed
yotsuda wants to merge 29 commits intoPowerShell:masterfrom
yotsuda:feature-convertto-json-factory-v2
Closed

Add PSJsonSerializerV2 for ConvertTo-Json using TruncatingConverterFactory#26654
yotsuda wants to merge 29 commits intoPowerShell:masterfrom
yotsuda:feature-convertto-json-factory-v2

Conversation

@yotsuda
Copy link
Copy Markdown
Contributor

@yotsuda yotsuda commented Jan 3, 2026

PR Summary

Implements TruncatingConverterFactory pattern for ConvertTo-Json V2 as discussed with @iSazonov in PR #26637.

This PR is an alternative implementation approach based on the Factory pattern, branched from feature-convertto-json-isazonov-approach.

Fixes #5749

PR Context

Background

In PR #26637, @iSazonov suggested implementing the Factory approach in a new PR to evaluate whether it provides a better architecture than the current implementation.

"I can't say for sure which approach will be the best for us in the end. Therefore, it is better to implement the factory approach with a new PR."

Thanks to @iSazonov's guidance on the technical considerations, this implementation now works correctly.

Solution

This implementation uses a TruncatingConverterFactory that dispatches to type-specific converters:

Type Converter
PSObject JsonConverterPSObject (Extended/Adapted properties)
IDictionary JsonConverterDictionary<T> (Base properties only)
IEnumerable JsonConverterEnumerable<T>
Composite objects JsonConverterComposite<T> (Base properties only)
Primitives STJ default converters

Architecture Benefits

  1. Type-level dispatch - STJ caches converters per type, improving performance
  2. Clear separation - Each converter handles one concern
  3. Extensibility - Easy to add new type-specific converters

Key Differences from PR #26637

Aspect PR #26637 This PR (Factory)
Raw object handling RawObjectWrapper class TruncatingConverterFactory dispatch
Type dispatch Manual in WriteValue Factory pattern with generic converters
Converter structure Single JsonConverterPSObject with _basePropertiesOnly flag Separate converters per type category

PR Checklist

  • PR has a meaningful title
  • Summarized changes
  • Make sure all .h, .cpp, .cs, .ps1 and .psm1 files have the correct copyright header
  • This PR is ready to merge
  • Breaking changes: Experimental feature needed
    • Experimental feature name: PSJsonSerializerV2
  • User-facing changes: Documentation needed
  • Testing: New tests added

Changes Made

Modified Files

ConvertToJsonCommandV2.cs

  • Added TruncatingConverterFactory - dispatches to appropriate converters based on type
  • Added JsonConverterDictionary<T> - handles IDictionary types
  • Added JsonConverterEnumerable<T> - handles IEnumerable types
  • Added JsonConverterComposite<T> - handles composite objects with Base properties only
  • Removed RawObjectWrapper class
  • Removed _basePropertiesOnly field and filterToPropertyOnly parameter from JsonConverterPSObject
  • Added WriteValue helper to JsonSerializerHelper

ConvertTo-Json.PSJsonSerializerV2.Tests.ps1

  • Removed array element test that now matches V1 behavior

Testing

Existing Tests (ConvertTo-Json.Tests.ps1)

  • 36 passed, 1 failed, 1 pending
  • The single failure is a DateTime test with timezone-dependent expectations (unrelated to this change)

V2 Specific Tests (ConvertTo-Json.PSJsonSerializerV2.Tests.ps1)

  • 4 passed, 0 failed

Related Work

PR/Issue Description
#26637 Original implementation (this PR is an alternative approach)
#5749 Dictionary with non-string keys handling

Branch Structure

gitGraph
    commit id: "master"
    branch system-text-json
    commit id: "77d8a1b0c"
    commit id: "159b76387"
    branch isazonov-approach
    commit id: "e7dfff962"
    commit id: "da0c8cbfb"
    branch factory-test
    commit id: "test" type: REVERSE
    checkout isazonov-approach
    commit id: "6913cb119"
    commit id: "2ecc51e05"
    commit id: "2c549207a" tag: "PR #26637"
    branch factory-v2
    commit id: "af393269b" tag: "HEAD"
    checkout system-text-json
    commit id: "07cea1dc5" tag: "PR #26624"
Loading
Branch Base PR Status
system-text-json master #26624 Superseded
isazonov-approach system-text-json #26637 Open
factory-test isazonov-approach - Experiment (failed)
factory-v2 isazonov-approach This PR ✅ Success

@yotsuda yotsuda marked this pull request as ready for review January 3, 2026 00:04
Copy link
Copy Markdown
Collaborator

@iSazonov iSazonov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yotsuda Thanks for starting the approach!

@yotsuda yotsuda force-pushed the feature-convertto-json-factory-v2 branch from 74a83c8 to 783f3b3 Compare January 4, 2026 05:03
@iSazonov
Copy link
Copy Markdown
Collaborator

iSazonov commented Jan 4, 2026

@yotsuda It seems you lost last commits - I don't see updates.

@yotsuda
Copy link
Copy Markdown
Contributor Author

yotsuda commented Jan 4, 2026

Copy link
Copy Markdown
Collaborator

@iSazonov iSazonov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yotsuda Great progress! Thanks!


// Convert to string when depth exceeded
string stringValue = LanguagePrimitives.ConvertTo<string>(pso.ImmediateBaseObjectIsEmpty ? pso : obj);
writer.WriteStringValue(stringValue);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems you don't understand my request. I mean if max depth is reached, V1 convert base object to string and process extended/adapted properties too. In V2 we only convert base object to string but don't process extended/adapted properties.

@iSazonov
Copy link
Copy Markdown
Collaborator

iSazonov commented Jan 5, 2026

Please see failed tests.
For "AddMember on JsonObject" I see:

$jstr
{
  "Major": 2,
  "Minor": 3,
  "Build": 4,
  "Revision": 14,
  "MajorRevision": 0,
  "MinorRevision": 14,
  "Note": "a version object",
  "Rev": 14,
  "IsOld": true
}

What is $jstr in V2?

As for 127 serialization I think we can select expected result based on the experimental feature status.

@iSazonov iSazonov added the CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log label Jan 5, 2026
Copy link
Copy Markdown
Collaborator

@iSazonov iSazonov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yotsuda Thanks for your efforts! I didn't expect that there would be so many difficulties, but it seems that we have already overcome most of them.


// GetTypeInfo() has internal caching, no need for additional cache
var typeInfo = JsonSerializerOptions.Default.GetTypeInfo(type);
return typeInfo.Kind == JsonTypeInfoKind.None;
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

public static bool ShouldSkipProperty(PSPropertyInfo prop)
{
// Check for Hidden attribute
if (prop.IsHidden)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting, how this (doesn't) works for script-defined classes.

@yotsuda
Copy link
Copy Markdown
Contributor Author

yotsuda commented Jan 6, 2026

Please see failed tests. For "AddMember on JsonObject" I see:
...
What is $jstr in V2?
As for 127 serialization I think we can select expected result based on the experimental feature status.

Both issues are fixed in the latest commits.

AddMember on JsonObject (Version + ETS):

V2 now produces the same output as V1 for Version objects with ETS properties:

$versionObject = New-Object System.Version 2, 3, 4, 14
$versionObject | Add-Member -MemberType NoteProperty -Name Note -Value "a version object"
$versionObject | Add-Member -MemberType ScriptProperty -Name Rev -Value { $this.Revision }
$versionObject | Add-Member -MemberType ScriptProperty -Name IsOld -Value { $this.Major -lt 3 }
$versionObject | ConvertTo-Json

Output (both V1 and V2):
{
  "Major": 2,
  "Minor": 3,
  "Build": 4,
  "Revision": 14,
  "MajorRevision": 0,
  "MinorRevision": 14,
  "Note": "a version object",
  "Rev": 14,
  "IsOld": true
}

The fix adds V1-compatible logic: "true primitive types" (string, DateTime, etc.) always serialize as scalars ignoring ETS, while non-primitive STJ scalar types (Version, IPAddress, etc.) serialize as objects when they have ETS properties.

[char]127 serialization:

Implemented as you suggested - the test now selects expected result based on experimental feature status:

$expectedToJson = $testCase.ToJson
if ($testCase.ToJsonV2 -and (Get-ExperimentalFeature PSJsonSerializerV2).Enabled)
{
    $expectedToJson = $testCase.ToJsonV2
}
  • V1: "" (DEL control character disappears)
  • V2: "\u007F" (correct Unicode escape)

Commit: 8efb723

private void SerializeAsObject(Utf8JsonWriter writer, PSObject pso, JsonSerializerOptions options)
{
writer.WriteStartObject();
AppendPSProperties(writer, pso, options, PSMemberViewTypes.Extended | PSMemberViewTypes.Adapted);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see V1 does:

  1. Enumerate properties using reflection.
  2. Add extended and adapted properties (excluding duplicates from 1.).

It seems if we request adapted properties we get reflection properties too. I guess there may be differences, but I do not know how it works in depth.

@SeeminglyScience @daxian-dbw Could you please help?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I created Version type object, then add ETS noteproperty with Add-Member, then request properties with PSMemberInfoIntegratingCollection:

  • for Base I see only the note property,
  • for Extended I see only the note property,
  • for Adapted I see the note property and native properties.

My understanding is that is how PSObject wrapping works. PSObject is a property bag. So for Base just the ps object is considered. For Extend - only ETS properties as exclusively separate. For Adapted - adapters are involved including .Net Adapter which return native .Net members. Some adapters can be invovled (including XML, CIM/WMI,COM). Property priority is Extended -> Adapded -> Base.

V1 code add native .Net properties by reflection then enumerate Extended and Adapted ones with PSMemberInfoIntegratingCollection and add them only if the name is not already in list. So reflection properties win.
If we use only PSMemberInfoIntegratingCollection in V2 - Extended properties win. It is breaking change.
Perhaps it would be worthwhile to adhere to this native PowerShell behavior, but since historically this cmdlet works differently, it is not difficult for us to do the same by taking properties from the Json Default cache or by psObj.Properties.Match("*", PSMemberTypes.Property); (although it is more expensive).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yotsuda Can you address my comment?

@yotsuda yotsuda requested a review from a team as a code owner January 7, 2026 01:59
@microsoft-github-policy-service microsoft-github-policy-service bot added the Review - Needed The PR is being reviewed label Jan 14, 2026
@iSazonov
Copy link
Copy Markdown
Collaborator

@yotsuda Please look failed test.

@microsoft-github-policy-service microsoft-github-policy-service bot removed the Review - Needed The PR is being reviewed label Jan 15, 2026
@yotsuda
Copy link
Copy Markdown
Contributor Author

yotsuda commented Jan 18, 2026

@iSazonov Thank you for pointing this out. I apologize for the delayed response to the CI test failures. I was running ConvertTo-Json.Tests.ps1 locally before commits, but I missed running Json.Tests.ps1, which contains the AddMember on JsonObject test.

CI Test Failures

Test File Platform Cause Added in this PR?
AddMember on JsonObject Json.Tests.ps1:231 All Version treated as scalar No (existing test)
Array of raw FileInfo ConvertTo-Json.Tests.ps1:330 Linux/macOS Property count 24 vs 28 Yes
Non-pure PSObject FileInfo ConvertTo-Json.Tests.ps1:432 Linux/macOS pwsh.exe path doesn't exist Yes
Version consistency ConvertTo-Json.Tests.ps1:359 All Pipeline vs InputObject mismatch Yes

Root Cause Analysis

The root cause of the AddMember on JsonObject failure is a mismatch between STJ's JsonTypeInfoKind classification and V1's explicit type handling:

Type STJ Kind V1 behavior V2 (before fix) Issue
string, DateTime, Guid None Scalar Scalar ✅ Match
System.Version None Object Scalar ❌ Mismatch

STJ returns Kind=None for types with registered JsonConverters (like Version), but V1's ProcessValue (JsonObject.cs:554-567) has an explicit type list that does NOT include Version, so V1 treats it as an object and enumerates its properties.

Background: Option C Implementation

In PR #26637, you suggested Option C to track raw vs PSObject instead of using a hardcoded type list. Based on this, I replaced the hardcoded IsPrimitiveType list with IsStjNativeScalarType() which uses JsonTypeInfoKind to dynamically detect scalar types.

However, STJ's JsonTypeInfoKind.None classification doesn't perfectly align with V1's behavior for all types. System.Version is one such case where STJ reports Kind=None (because it has a registered JsonConverter), but V1 treats it as an object.

Two Possible Fixes

Fix#1: STJ-based with Exception List

if (type == typeof(Version) || type == typeof(Type))
    return false;  // Treat as object, not scalar
return typeInfo.Kind == JsonTypeInfoKind.None;
  • Pros: Maintains STJ-based design (Option C spirit), minimal code change
  • Cons: Risk of undiscovered mismatches with other types

Fix#2: Revert to V1's Explicit Type List

return obj is string or char or bool or DateTime or DateTimeOffset 
    or Guid or Uri or double or float or decimal or BigInteger;
  • Pros: Guarantees V1 compatibility
  • Cons: Loses STJ-based classification benefits, reverts the Option C direction we discussed

Local Test Results (Fix#1)

I tested Fix#1 locally:

File Passed Failed Notes
Json.Tests.ps1 188 1 AddMember on JsonObject ✅ passes. The 1 failure is a datetime locale issue (unrelated to V2)
ConvertTo-Json.Tests.ps1 44 1 Version consistency fails (test added in this PR)
V2-specific tests 2 0 All pass

Proposed Next Steps

  1. Apply Fix#1 (or Fix#2 based on your feedback)
  2. Remove or fix the tests I added in this PR that fail on Linux/macOS

Which fix do you prefer, or do you have a different solution in mind?

@iSazonov
Copy link
Copy Markdown
Collaborator

@yotsuda Thanks!

We need to strive for backward compatibility to make it easier to migrate to an updated cmdlet. The same tests should work for both versions.
We already have one exclusion for object type so we could do the same for Version type.
But there is second place for possible fix. There is specific behavior for string and Datetime. Perhaps the fix should be that we create an exclusion list of string, Datetime, Version.

@yotsuda
Copy link
Copy Markdown
Contributor Author

yotsuda commented Jan 20, 2026

@iSazonov Thank you for the feedback!

I've applied Fix#1 (Version exception) and fixed the cross-platform test issues in commit 6984cf3.

However, I'd like to clarify the difference between string/DateTime and Version handling:

Current Implementation

The code has two separate exclusion mechanisms for different purposes:

Method Types Purpose
IsV1EtsSkipType string, DateTime Skip ETS properties (V1 ignores ETS on these)
IsStjNativeScalarType object, Version Treat as non-scalar (serialize as object with properties)

V1 Behavior Verification

# string/DateTime: ETS properties are IGNORED
$str = "hello"
$str | Add-Member -NotePropertyName ETS -NotePropertyValue "test" -PassThru | ConvertTo-Json
# Output: "hello"  (no ETS)

# Version: ETS properties are INCLUDED
$ver = [Version]"1.2.3"
$ver | Add-Member -NotePropertyName ETS -NotePropertyValue "test" -PassThru | ConvertTo-Json -Compress
# Output: {"Major":1,"Minor":2,"Build":3,"Revision":-1,...,"ETS":"test"}

Why Not Combine Into One List?

If we added Version to IsV1EtsSkipType (alongside string/DateTime), ETS properties on Version objects would be skipped, breaking backward compatibility.

The current fix correctly handles Version by:

  1. Returning false from IsStjNativeScalarType → treats Version as non-scalar
  2. NOT adding it to IsV1EtsSkipType → preserves ETS properties

This matches V1 behavior exactly.

Changes in This Commit (6984cf3)

  1. Fix#1 applied: Added Version to IsStjNativeScalarType exception list
  2. Cross-platform test fixes:
    • Changed pwsh.exeSystem.Management.Automation.dll (exists on all platforms)
    • Changed property count check to platform-specific (Windows: 24, Unix: 28)
  3. Culture-independent datetime test: Added en-US culture wrapper for locale-dependent assertions in Json.Tests.ps1

Question: V2-Specific Tests

I've created ConvertTo-Json.PSJsonSerializerV2.Tests.ps1 to document V2-specific behaviors that differ from V1 (e.g., non-string dictionary keys support).

If your goal is to make V2 fully compatible with V1, I should:

  1. Modify the code to match V1 behavior for these cases
  2. Remove the V2-specific tests

Would you like me to pursue full V1 compatibility, or is it acceptable for V2 to have documented behavioral improvements while maintaining compatibility for common use cases?

@iSazonov
Copy link
Copy Markdown
Collaborator

@yotsuda Thanks for clarify! It was my wrong understanding of Version behavior.

I feel there is another issue. If a type has a custom converter, it return JsonTypeInfoKind.None then we consider it as scalar type and call the type converter without out custom depth control. Yes?

If your goal is to make V2 fully compatible with V1, I should:

I am ok with "non-string dictionary keys support" added.
Version behavior you already fixed.
Are there other breaking changes you mean?

@yotsuda
Copy link
Copy Markdown
Contributor Author

yotsuda commented Jan 20, 2026

@iSazonov Thank you for the clarification!

Custom Converter + Kind=None

If a type has a custom converter, it return JsonTypeInfoKind.None then we consider it as scalar type and call the type converter without out custom depth control. Yes?

Yes, that's the current behavior. For types with Kind=None:

  • If a dedicated converter exists (Double, Float, Type), it handles serialization
  • Otherwise, STJ's default serialization is used
  • In both cases, no depth control is applied

My understanding is that this is acceptable because these types are "leaf" values:

  • Double/Float: PSJsonDoubleConverter outputs a number or string ("Infinity")
  • Type: PSJsonTypeConverter outputs AssemblyQualifiedName string

Note: BigInteger has Kind=Object, but PSJsonBigIntegerConverter is registered first, so it's handled before reaching the factory.

If you see any issues with this approach, please let me know.

Other Breaking Changes

Besides the non-string dictionary keys support, I'm not currently aware of other breaking changes, but I may have missed something.

If you have specific scenarios you'd like me to verify, I'd be happy to test them.

@iSazonov
Copy link
Copy Markdown
Collaborator

If you see any issues with this approach, please let me know.

If an user creates custom class with custom converter the class will not be serialized by our reflection converter with our depth control.

@yotsuda
Copy link
Copy Markdown
Contributor Author

yotsuda commented Jan 20, 2026

@iSazonov Thank you for the clarification.

You're right - custom converters bypass our depth control. I investigated how V1 handles this.

V1's Approach

V1 never uses [JsonConverter] attributes. Looking at the code:

  1. ConvertToJson calls ProcessValue to pre-process the object
  2. ProcessValue converts objects to Dictionary<string, object> using reflection (ProcessCustomObject<JsonIgnoreAttribute>)
  3. The converted dictionary is then passed to JsonConvert.SerializeObject

Since the original object is converted to a dictionary before being passed to Newtonsoft.Json, any [JsonConverter] attribute on the class is ignored.

Options

  1. Keep current (V2 improvement): Allow custom converters to work, accepting that they bypass depth control. Users who define [JsonConverter] attributes likely understand and expect this behavior.

  2. Match V1: Ignore [JsonConverter] attributes, force reflection-based serialization with depth control.

    • Technically feasible: Types with [JsonConverter] return Kind=None from JsonTypeInfo
    • Implementation: Add check in CanConvert:
      if (typeToConvert.IsDefined(typeof(JsonConverterAttribute), inherit: true))
      {
          return true;
      }

Which approach do you prefer?

@iSazonov
Copy link
Copy Markdown
Collaborator

@yotsuda I think it's time to take stock. I feel my expectations that the new code will be simpler, faster and the functionality extensible have not been fulfilled. By creating workarounds, we have lost all the benefits of STJ.
The main problem is depth control.
I think it makes sense to discuss this in .Net Runtime repository.

@yotsuda
Copy link
Copy Markdown
Contributor Author

yotsuda commented Jan 21, 2026

@iSazonov I really appreciate all your detailed reviews and guidance throughout this PR.

I agree that maintaining V1 compatibility while keeping V2 simple, fast, and extensible has proven to be a difficult balance. Here's a summary of the challenges we encountered:

Challenge Root Cause Solution Trade-off (Lost V2 Benefit)
Types with problematic properties STJ's default converter fails on types like FileInfo (circular refs, throwing properties) Manual PSObject property enumeration instead of STJ reflection Cannot leverage STJ's native object serialization
Depth control STJ's MaxDepth throws on overflow; V1 converts to string Custom converters with manual depth tracking Overhead of custom converters vs STJ native serialization
Reflection properties must win over ETS PSMemberInfoIntegratingCollection prioritizes Extended properties Enumerate via JsonTypeInfo first, then append ETS with duplicate exclusion Two-pass enumeration; added complexity
ETS on arrays/enumerables PSObject.AsPSObject() doesn't preserve ETS from original wrapper Detect ETS and serialize as {"value":[...],"MyProp":...} Special-case handling
ETS on scalar types STJ serializes scalars directly without object wrapper Wrap as {"value":scalar,"MyProp":...} when ETS present Special-case handling for DBNull, NullString, Uri, etc.
Version/Type as objects STJ treats these as Kind=None (scalar) Explicit check in IsScalar to force reflection path Hard-coded type exceptions
Custom [JsonConverter] attributes V1 ignores them; V2/STJ uses them, bypassing depth control Unresolved Trade-off between V1 compatibility and STJ extensibility

Intentional improvement in V2: Non-string dictionary keys are now supported (V1 threw an error).

As you noted, these workarounds have added complexity. I hope this code and analysis can serve as useful material for the .NET Runtime discussion. Please let me know if there's anything you'd like me to revise.

@iSazonov
Copy link
Copy Markdown
Collaborator

@yotsuda Thanks! I opened dotnet/runtime#123424

@iSazonov
Copy link
Copy Markdown
Collaborator

@yotsuda dotnet/runtime#123424 was closed. There are huge limitations in STJ design and implementation, it's incredible that they'll change anything in the near future. We have to close the PR.
So we will improve the cmdlet code that we have.
You can work on it if you wish. First of all, on backward compatibility tests.

Thanks for your efforts!

@yotsuda
Copy link
Copy Markdown
Contributor Author

yotsuda commented Jan 27, 2026

@iSazonov Thanks for the clarification and for filing the upstream issue.

I understand STJ migration is not feasible given the design limitations. Closing this PR.

I'll continue with #26639 for compatibility tests. The test failures there are fixed and it's ready for review. Please share your thoughts on additional tests or cmdlet improvements when you have time.

Thanks for your guidance throughout this effort!

@yotsuda yotsuda closed this Jan 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Parameter to ConvertTo-Json to ignore unsupported properties

2 participants