Output custom object with headers when no values are present by jshigetomi · Pull Request #25096 · PowerShell/PowerShell

jshigetomi · 2025-02-26T16:30:35Z

PR Summary

This pull request includes changes to improve the handling of empty and null values in the ConvertFrom-Csv command and updates the corresponding unit tests to verify the new behavior. The most important changes include modifying the Import method in CsvCommands.cs and adding new unit tests in ConvertFrom-Csv.Tests.ps1.

Enhancements to ConvertFrom-Csv command:

src/Microsoft.PowerShell.Commands.Utility/commands/utility/CsvCommands.cs: Updated the Import method to skip empty lines only when using Import-CSV, but to keep them when using ConvertFromCSV.

Unit tests updates:

test/powershell/Modules/Microsoft.PowerShell.Utility/ConvertFrom-Csv.Tests.ps1: Added new unit tests to verify that ConvertFrom-Csv handles empty and null values correctly.

PR Context

Improve CSV parsing edge case handling and add comprehensive unit tests

PR Checklist

…wershell into fixNoValueCSVInput

jshigetomi · 2025-02-26T21:07:26Z

I've adjusted the behavior of ConvertFrom-CSV according to the discussion on issue #24698. Essentially I removed the feature that skips blank lines in the Import method in ImportCsvHelper. It still keeps the last cell null so as to keep this from being a breaking change. #17702

surfingoldelephant · 2025-02-27T01:23:11Z

src/Microsoft.PowerShell.Commands.Utility/commands/utility/CsvCommands.cs

+                if (values.Count == 1 && string.IsNullOrEmpty(values[0]) && !(_cmdlet is ConvertFromCsvCommand))
                {
-                    // skip the blank lines
+                    // Skip empty lines when Import-CSV but keep empty lines when ConvertFromCSV


The bug (#24698) affects both ConvertFrom-Csv and Import-Csv; I think the fix should be applied to both commands for consistency. ConvertFrom-Csv was used throughout the original issue for demonstration purposes only.

Do we know if there's a specific reason why the check for values.Count == 1 && string.IsNullOrEmpty(values[0]) was included in the first place?

This comes from Windows PowerShell.

When I disable it for Import-CSV as well, `n's are captured as an extra element. Is this the behavior we want?

Describe "Import-Csv with different newlines" -Tags "CI" { It "Test import-csv with '<name>' newline" -TestCases @( @{ name = "CR"; newline = "`r" } @{ name = "LF"; newline = "`n" } @{ name = "CRLF"; newline = "`r`n" } ) { param($newline) $csvFile = Join-Path $TestDrive -ChildPath $((New-Guid).Guid) $delimiter = ',' "h1,h2,h3$($newline)11,12,13$($newline)21,22,23$($newline)" | Out-File -FilePath $csvFile $returnObject = Import-Csv -Path $csvFile -Delimiter $delimiter $returnObject.Count | Should -Be 2 $returnObject[0].h1 | Should -Be 11 $returnObject[0].h2 | Should -Be 12 $returnObject[0].h3 | Should -Be 13 $returnObject[1].h1 | Should -Be 21 $returnObject[1].h2 | Should -Be 22 $returnObject[1].h3 | Should -Be 23 } }

Is this the behavior we want?

No, it isn't.

This is more about ensuring whatever fix is chosen for #24698 is applied to both ConvertFrom-Csv and Import-Csv. In other words, the example CSV data below should deserialize consistently, irrespective of the source being a file or a string.

"P1" "" "" ""

Can #24698 be fixed for both commands without introducing the newline issue?

I'm thinking it can be deserialized consistently but would result in just one empty string value per header.
For example,

"P1" "" "" "" # Would produce [pscustomobject] @{ P1 = '' } ignoring the other empty cells

@' "P1","P2" "", "", '@ | ConvertFrom-Csv # Would produce [pscustomobject] @{ P1 = ''; P2 = '' } ignoring the other empty cells again

This way we can produce a header only output consistently.

Thoughts?

That's not what I had in mind when I filed the issue. For each non-empty line in input data, an object should be emitted by ConvertFrom-Csv/Import-Csv.

The following (using either command) produces 3 objects:

@' "P1" "a" "b" "c" '@ | ConvertFrom-Csv

Therefore, the following (using either command) should also produce 3 objects:

@' "P1" "" "" "" '@ | ConvertFrom-Csv

Likewise, this produces 2 objects, because the empty line is correctly ignored:

@' "P1" "a" "c" '@ | ConvertFrom-Csv

Therefore, this should also produce 2 objects:

@' "P1" "" "" '@ | ConvertFrom-Csv

iSazonov · 2025-02-28T19:32:35Z

@jshigetomi Root of the issue in ParseNextRecord method. It does not distinguish between an input string without characters and a string with two double quotes "".

PowerShell/src/Microsoft.PowerShell.Commands.Utility/commands/utility/CsvCommands.cs

Lines 1609 to 1613 in bfdc498

    
           result.Add(current.ToString()); 
        
           current.Remove(0, current.Length); 
        
           // New line outside quote is end of word and end of record 
        
           break;

Here current.ToString() returns string.Empty for these two cases. We can check current.Length > 0 before call ToString(). I suggest the fix (with removing

PowerShell/src/Microsoft.PowerShell.Commands.Utility/commands/utility/CsvCommands.cs

Lines 1390 to 1394 in bfdc498

    
           if (values.Count == 1 && string.IsNullOrEmpty(values[0]) && !(_cmdlet is ConvertFromCsvCommand)) 
        
           { 
        
               // Skip empty lines when Import-CSV but keep empty lines when ConvertFromCSV 
        
               continue; 
        
           }

)

But we get to Zugzwang - sometimes we want to, sometimes we don't. Of course, it would be possible to add more conditions, but this would require a more significant code change in same methods. My suggestion is not to complicate the code. If there are concerns, then make this change an experimental feature.

iSazonov · 2025-03-01T13:46:10Z

@jshigetomi I checked my guess. I had to make a few more changes. You can see the result here:

master...iSazonov:PowerShell:csvblank

jshigetomi · 2025-03-03T19:14:00Z

@iSazonov I'm seeing test failure for Import-CSV

Maybe a warning to the user would be better for no value csvs?

iSazonov · 2025-03-03T19:25:44Z

I don't see errors in my branch.

jshigetomi · 2025-03-03T22:15:48Z

Hmm I'm having some trouble reproducing your results.

master...iSazonov:PowerShell:csvblank#diff-2b77454e7d234e46e6ebdb6c1be4ac0cb9c02b88bbe0e61c9acdfa3f66491e81R1383-R1390

How are you resolving the brackets here?

iSazonov · 2025-03-04T12:33:43Z

@jshigetomi Please look #25120

jshigetomi · 2025-03-06T20:32:52Z

/azp run

azure-pipelines · 2025-03-06T20:33:07Z

Azure Pipelines successfully started running 2 pipeline(s).

jshigetomi · 2025-03-27T20:26:52Z

@surfingoldelephant @iSazonov Thanks for the pointer for Set-Content. I was not aware of the newline behavior.

I made the ConvertFrom-CSV and Import-CSV tests the same and both behave the same.

iSazonov · 2025-03-28T03:22:28Z

test/powershell/Modules/Microsoft.PowerShell.Utility/Import-Csv.Tests.ps1

+
+Describe "Import-Csv with empty and null values" {
+
+    Context 'Empty CSV Fields' {


Since the PR is about edge cases I think about additional tests:

only header presents without newline

header presents with one empty newline

And more:

empty file (-Header parameter presents)

file with one empty line (-Header parameter presents)

Just to be clear, we're expecting those tests to produce no output (zero custom objects).

Let's also include Import-Csv/ConvertFrom-Csv tests with multiple empty lines.

Header only

@' P1,P2 '@ | ConvertFrom-Csv

Header followed by one empty line

@' P1,P2 '@ | ConvertFrom-Csv

Header followed by multiple empty lines

@' P1,P2 '@ | ConvertFrom-Csv

Empty input and -Header specified

'' | ConvertFrom-Csv -Header P1, P2

One empty line and -Header specified

@' '@ | ConvertFrom-Csv -Header P1, P2

Multiple empty lines and -Header specified

@' '@ | ConvertFrom-Csv -Header P1, P2

Perhaps additional -Header tests to cover the fix:

@' , A1,A2 B1,B2 , '@ | ConvertFrom-Csv -Header P1, P2 # Expected: [pscustomobject] @{ P1 = ''; P2 = '' } [pscustomobject] @{ P1 = 'A1'; P2 = 'A2' } [pscustomobject] @{ P1 = 'B1'; P2 = 'B2' } [pscustomobject] @{ P1 = ''; P2 = $null }

surfingoldelephant

I think we also need to take a look at no detected delimiter scenarios.

# Detected delimiter:

@'
P1,P2,P3
A1,,
B1,,
C1,,
'@ | ConvertFrom-Csv

# Current/expected behavior:
[pscustomobject] @{ P1 = 'A1'; P2 = ''; P3 = '' }
[pscustomobject] @{ P1 = 'B1'; P2 = ''; P3 = '' }
[pscustomobject] @{ P1 = 'C1'; P2 = ''; P3 = $null }

# No detected delimiter:

@'
P1,P2,P3
A1
B1
C1
'@ | ConvertFrom-Csv

# Current/expected behavior:
[pscustomobject] @{ P1 = 'A1'; P2 = $null; P3 = $null }
[pscustomobject] @{ P1 = 'B1'; P2 = $null; P3 = $null }
[pscustomobject] @{ P1 = 'C1'; P2 = $null; P3 = $null }


@'
P1;P2;P3
A1
B1
C1
'@ | ConvertFrom-Csv -Delimiter ';'

# Current/expected behavior:
[pscustomobject] @{ P1 = 'A1'; P2 = $null; P3 = $null }
[pscustomobject] @{ P1 = 'B1'; P2 = $null; P3 = $null }
[pscustomobject] @{ P1 = 'C1'; P2 = $null; P3 = $null }

Notice when there is no (detected) delimiter, the resultant value is always $null. Can we add tests for this?

…wershell into fixNoValueCSVInput

jshigetomi · 2025-08-27T19:44:29Z

/azp run

azure-pipelines · 2025-08-27T19:44:41Z

Azure Pipelines successfully started running 1 pipeline(s).

jshigetomi · 2025-09-29T18:51:06Z

/azp run PowerShell-Windows-Packaging-CI, PowerShell-CI-linux-packaging

azure-pipelines · 2025-09-29T18:51:14Z

Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

Copilot

Pull request overview

This pull request improves the handling of empty and null values in CSV parsing commands (Import-Csv and ConvertFrom-Csv). The changes ensure that both cmdlets properly distinguish between truly empty lines (which should be skipped) and lines containing only delimiters (which represent rows with empty field values and should be processed).

Changes:

Improved logic for detecting and handling blank lines in CSV parsing
Added comprehensive test coverage for edge cases involving empty fields, null values, and various delimiter scenarios
Refactored existing tests to reduce duplication and improve maintainability
Fixed a spelling error in a test variable name

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/Microsoft.PowerShell.Commands.Utility/commands/utility/CsvCommands.cs	Updated Import method to correctly identify blank lines (values.Count == 0) instead of incorrectly treating single empty strings as blank lines. Changed ParseNextRecord to return instead of break for clarity and added logic to prevent empty results from being added at the start of records. Changed loop condition from while(true) to while(!EOF) for clarity.
test/powershell/Modules/Microsoft.PowerShell.Utility/Import-Csv.Tests.ps1	Fixed spelling of variable name (emptyValueCsv). Refactored duplicate test cases into a single loop-based test for better maintainability. Added extensive new test coverage for empty/null value handling across multiple contexts: empty CSV fields, header-only scenarios, empty input with -Header parameter, edge cases with whitespace and special characters, type information handling, delimiter edge cases, and large data scenarios.
test/powershell/Modules/Microsoft.PowerShell.Utility/ConvertFrom-Csv.Tests.ps1	Added comprehensive test coverage matching the Import-Csv tests, covering the same edge cases but using ConvertFrom-Csv instead of Import-Csv, ensuring both cmdlets handle empty and null values consistently.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

test/powershell/Modules/Microsoft.PowerShell.Utility/ConvertFrom-Csv.Tests.ps1

…om-Csv.Tests.ps1 Co-authored-by: Copilot <[email protected]>

surfingoldelephant · 2026-01-21T05:45:24Z

test/powershell/Modules/Microsoft.PowerShell.Utility/ConvertFrom-Csv.Tests.ps1

+        It 'Should handle no delimiter with empty fields' {
+            $result = @'
+P1,P2,P3
+A1
+B1
+C1
+'@ | ConvertFrom-Csv
+
+            $result.Count | Should -Be 3
+            $result[0].P1 | Should -Be 'A1'
+            $result[0].P2 | Should -BeNullOrEmpty
+            $result[0].P3 | Should -BeNullOrEmpty
+            $result[1].P1 | Should -Be 'B1'
+            $result[1].P2 | Should -BeNullOrEmpty
+            $result[1].P3 | Should -BeNullOrEmpty
+            $result[2].P1 | Should -Be 'C1'
+            $result[2].P2 | Should -BeNullOrEmpty
+            $result[2].P3 | Should -BeNullOrEmpty
+        }
+    }
+}


I think we need to watch out for -BeNullOrEmpty. That returns $true for both $null and empty strings (''), whereas most of the tests should be checking explicitly for $null or an empty string, not both.

For example, in the quoted test, P2 and P3 should only be $null, yet the test will pass if the value is ''.

I suggest replacing all instances of -BeNullOrEmpty to either -BeExactly $null or -BeExactly '' depending on the expected result.

Perhaps also update existing -Be usage to -BeExactly. It won't change the behavior in this case, but signals clearer intent.

Justin Chung and others added 9 commits February 26, 2025 10:26

Output custom object with headers when no values are present

df47ac8

Merge branch 'master' into fixNoValueCSVInput

4ccc9ad

Remove if statement that skips blank lines

ba25fa7

Merge branch 'master' into fixNoValueCSVInput

7ef2c8b

Only remove the if statement for ConvertFrom-CSV

83a53ef

Merge branch 'fixNoValueCSVInput' of https://github.com/powershell/po…

b7f491d

…wershell into fixNoValueCSVInput

Add test for testing null values and empty string values

3786536

Add comment about skiping lines

50db112

Delete trailing whitespace

c4950da

PowerShell deleted a comment from azure-pipelines bot Feb 26, 2025

jshigetomi mentioned this pull request Feb 26, 2025

CSV deserialization outputs nothing if there are only a certain number of empty fields #24698

Open

5 tasks

Add more test cases for empty and null values

472c6d0

PowerShell deleted a comment from azure-pipelines bot Feb 26, 2025

Fix test to make it pester 4.10 compatible

bfdc498

surfingoldelephant reviewed Feb 27, 2025

View reviewed changes

Add iSazonov's changes

64e6de8

Justin Chung added 2 commits March 6, 2025 11:33

Remove skip no values check and add skip blank check

8dd4a66

Revert to private feed

6c2c6c7

PowerShell deleted a comment from azure-pipelines bot Mar 6, 2025

iSazonov reviewed Mar 28, 2025

View reviewed changes

surfingoldelephant reviewed Mar 31, 2025

View reviewed changes

Added tests for edge cases and updated old test for Context and typo

6e3fa23

jshigetomi requested review from iSazonov and surfingoldelephant August 4, 2025 18:32

jshigetomi added 6 commits August 4, 2025 14:20

adjust import-csv tests

de52856

Adjust ConvertFrom csv tests

87c5def

fix header param test

1ba4e61

Merge branch 'master' into fixNoValueCSVInput

7a24009

Clean up comments

effd7b2

Merge branch 'fixNoValueCSVInput' of https://github.com/powershell/po…

5bd8b40

…wershell into fixNoValueCSVInput

Merge branch 'master' into fixNoValueCSVInput

db0bbea

jshigetomi added 2 commits September 2, 2025 11:49

Merge branch 'master' into fixNoValueCSVInput

f693967

Merge branch 'master' into fixNoValueCSVInput

0b22c87

jshigetomi closed this Jan 13, 2026

github-project-automation bot moved this from PR In Progress to Done in PowerShell Team Reviews/Investigations Jan 13, 2026

jshigetomi reopened this Jan 13, 2026

Make tests platform aware

903393f

Copilot AI review requested due to automatic review settings January 13, 2026 19:02

Copilot started reviewing on behalf of jshigetomi January 13, 2026 19:02 View session

Copilot AI reviewed Jan 13, 2026

View reviewed changes

test/powershell/Modules/Microsoft.PowerShell.Utility/ConvertFrom-Csv.Tests.ps1 Outdated Show resolved Hide resolved

jshigetomi and others added 2 commits January 13, 2026 13:25

Update test/powershell/Modules/Microsoft.PowerShell.Utility/ConvertFr…

f1001ab

…om-Csv.Tests.ps1 Co-authored-by: Copilot <[email protected]>

Merge branch 'master' into fixNoValueCSVInput

bc9f972

surfingoldelephant approved these changes Jan 21, 2026

View reviewed changes

surfingoldelephant reviewed Jan 21, 2026

View reviewed changes


		Describe "Import-Csv with empty and null values" {

		Context 'Empty CSV Fields' {

Conversation

jshigetomi commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Context

PR Checklist

Uh oh!

jshigetomi commented Feb 26, 2025

Uh oh!

surfingoldelephant Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

iSazonov Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

jshigetomi Feb 27, 2025

Choose a reason for hiding this comment

Uh oh!

surfingoldelephant Feb 28, 2025

Choose a reason for hiding this comment

Uh oh!

jshigetomi Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surfingoldelephant Feb 28, 2025

Choose a reason for hiding this comment

Uh oh!

iSazonov commented Feb 28, 2025

Uh oh!

iSazonov commented Mar 1, 2025

Uh oh!

jshigetomi commented Mar 3, 2025

Uh oh!

iSazonov commented Mar 3, 2025

Uh oh!

jshigetomi commented Mar 3, 2025

Uh oh!

iSazonov commented Mar 4, 2025

Uh oh!

jshigetomi commented Mar 6, 2025

Uh oh!

azure-pipelines bot commented Mar 6, 2025

Uh oh!

jshigetomi commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iSazonov Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

surfingoldelephant Mar 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surfingoldelephant left a comment

Choose a reason for hiding this comment

Uh oh!

jshigetomi commented Aug 27, 2025

Uh oh!

azure-pipelines bot commented Aug 27, 2025

Uh oh!

jshigetomi commented Sep 29, 2025

Uh oh!

azure-pipelines bot commented Sep 29, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

surfingoldelephant Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

jshigetomi commented Feb 26, 2025 •

edited

Loading

jshigetomi Feb 28, 2025 •

edited

Loading

jshigetomi commented Mar 27, 2025 •

edited

Loading

surfingoldelephant Mar 31, 2025 •

edited

Loading

surfingoldelephant Jan 21, 2026 •

edited

Loading