Relax RandomAccess type requirements: make Read/Write methods work with non-seekable files#125512
Relax RandomAccess type requirements: make Read/Write methods work with non-seekable files#125512
Conversation
…seekable files - Add native readv/writev sys-call wrappers in pal_io.c and pal_io.h - Add native entrypoints for ReadV/WriteV in entrypoints.c - Add managed interop files Interop.ReadV.cs and Interop.WriteV.cs - Add interop references to System.Private.CoreLib.Shared.projitems - Modify RandomAccess.cs - relax ValidateInput to allow unseekable handles - Update XML docs for Read/Write methods (remove NotSupportedException, note offset ignored) - Modify RandomAccess.Unix.cs - use readv/writev for non-seekable scatter/gather - Update Base.cs - remove ThrowsNotSupportedExceptionForUnseekableFile test - Add NonSeekable.cs tests using SafeFileHandle.CreateAnonymousPipe - Add NonSeekable_AsyncHandles.cs tests - Update test .csproj to include new test files Co-authored-by: adamsitnik <[email protected]>
…d of AnonymousPipeServerStream Co-authored-by: adamsitnik <[email protected]>
… consistency Co-authored-by: adamsitnik <[email protected]>
adamsitnik
left a comment
There was a problem hiding this comment.
@copilot please address my feedback
src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Unix.cs
Outdated
Show resolved
Hide resolved
src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Unix.cs
Outdated
Show resolved
Hide resolved
…ndling - ReadScatterAtOffset: try PReadV first, fall back to ReadV on ENXIO/ESPIPE - WriteGatherAtOffset: try PWriteV first, fall back to WriteV on ENXIO/ESPIPE - SystemNative_ReadV: handle EAGAIN/EWOULDBLOCK with poll loop (like ReadFromNonblocking) - SystemNative_WriteV: handle EAGAIN/EWOULDBLOCK with poll loop (like WriteToNonblocking) Co-authored-by: adamsitnik <[email protected]>
adamsitnik
left a comment
There was a problem hiding this comment.
@copilot please address my feedback
@copilot use the RandomAccess.Write in following places (respect the current EPIPE handling by swallowing specific exception):
runtime/src/libraries/System.Console/src/System/ConsolePal.Browser.cs
Lines 40 to 61 in 9a2c40b
runtime/src/libraries/System.Console/src/System/ConsolePal.Unix.cs
Lines 981 to 1030 in 9a2c40b
use the RandomAccess.Read in following places:
runtime/src/libraries/System.Console/src/System/ConsolePal.Wasi.cs
Lines 251 to 259 in 9a2c40b
runtime/src/libraries/System.Console/src/System/ConsolePal.Unix.cs
Lines 946 to 954 in 9a2c40b
If it does not break the build, remove related references from the project file (the list below is not complete, please verify it on your own):
runtime/src/libraries/System.Console/src/System.Console.csproj
Lines 84 to 89 in 9a2c40b
runtime/src/libraries/System.Console/src/System.Console.csproj
Lines 236 to 239 in 9a2c40b
src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Unix.cs
Outdated
Show resolved
Hide resolved
src/libraries/System.Runtime/tests/System.IO.FileSystem.Tests/RandomAccess/NonSeekable.cs
Outdated
Show resolved
Hide resolved
…ack, use RandomAccess in Console - Extract NeedsNonOffsetFallback helper method in RandomAccess.Unix.cs - Improve tests: deduplicate cancellation checks, handle Read returning 0 in loops, use AssertExtensions.SequenceEqual - Use RandomAccess.Read/Write in ConsolePal.Browser.cs, ConsolePal.Unix.cs, ConsolePal.Wasi.cs - Remove unused Interop.Read.cs, Interop.Write.cs, Interop.Poll.cs references from System.Console.csproj Co-authored-by: adamsitnik <[email protected]>
Done in commit a5d8cb2:
All 9741 |
adamsitnik
left a comment
There was a problem hiding this comment.
@copilot please address my feedback
src/libraries/System.Runtime/tests/System.IO.FileSystem.Tests/RandomAccess/NonSeekable.cs
Outdated
Show resolved
Hide resolved
src/libraries/System.Runtime/tests/System.IO.FileSystem.Tests/RandomAccess/NonSeekable.cs
Outdated
Show resolved
Hide resolved
| child2.Start(); | ||
| char[] output = new char[6]; | ||
| child2.StandardOutput.Read(output, 0, output.Length); | ||
| child2.StandardOutput.ReadBlock(output, 0, output.Length); |
There was a problem hiding this comment.
Wasn't this WaitForPeerProcess always wrong, and we were just getting lucky because of implementation detail? i.e. regardless of that debug condition, there's no guarantee Read would always return as much as was requested
src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Unix.cs
Outdated
Show resolved
Hide resolved
src/libraries/System.Console/src/System/ConsolePal.Unix.ConsoleStream.cs
Show resolved
Hide resolved
adamsitnik
left a comment
There was a problem hiding this comment.
@copilot please address the feedback where I've tagged you
| child2.Start(); | ||
| char[] output = new char[6]; | ||
| child2.StandardOutput.Read(output, 0, output.Length); | ||
| child2.StandardOutput.ReadBlock(output, 0, output.Length); |
There was a problem hiding this comment.
Yes, more or less we were always lucky so far. I wanted to be clear why it was not needed before (to show it's not a bug now)
src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Unix.cs
Outdated
Show resolved
Hide resolved
… calls, rename helper Co-authored-by: adamsitnik <[email protected]>
…d/Write methods Co-authored-by: adamsitnik <[email protected]>
…ccess-requirements
Description
Relaxes
System.IO.RandomAccessto support non-seekable handles (pipes, sockets, character devices) by falling back to non-offset syscalls when the file does not support seeking. Updates affected call sites inSystem.ConsoleandSystem.Diagnostics.Processto useRandomAccess.Read/Write, and adds comprehensive tests for non-seekable handle scenarios.Core changes
RandomAccessvalidation to allow non-seekable handles forRead*/Write*methods (while keeping seek-only behavior forGetLength/SetLength)SystemNative_ReadV/SystemNative_WriteVnative exports withEAGAIN/EWOULDBLOCKpoll-loop handling andGetAllowedVectorCountcapping forIOV_MAXShouldFallBackToNonOffsetSyscallhelper for ENXIO/ESPIPE fallback logic, with consolidated control flow to avoid duplicate syscallsNotSupportedExceptiondocs qualified with "In .NET 10 and earlier versions, ..." for backward compatibilityConsole/Process call site updates
Interop.Sys.Read/WritewithRandomAccess.Read/WriteinConsolePal.Unix.cs,ConsolePal.Wasi.cs, andConsolePal.Unix.ConsoleStream.cs(with EPIPE handling preserved)ConsolePal.Browser.cschanges (WASM does not supportlibSystem.Nativedynamic linking)foreachloop inUpdatedCachedCursorPositioninstead of index-based iterationProcessWaitingTeststo useReadBlockinstead ofReadto handle partial reads correctlyTests
AssertCanceledhelper,AssertExtensions.SequenceEqual, break onread == 0PartialReadssync/async tests into a single[Theory]Testing
System.IO.FileSystem.Testspass (9741 tests)System.Consolebuilds for all 8 targets (unix, windows, browser, wasi, android, ios, tvos, default)ProcessWaitingTestspass withReadBlockfixOriginal prompt
This section details on the original issue you should resolve
<issue_title>Relax RandomAccess type requirements, make all Read*|Write* methods work with non-seekable files</issue_title>
<issue_description>In .NET 6 we have introduced new type called
RandomAccessthat allows for reading and writing to specific file offset.As of today, all it's
Read*andWrite*methods throw when given handle points to a non-seekable file like socket or pipe:runtime/src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.cs
Line 251 in d033e1a
But it's not a problem for it's internal implementation (used by
FileStream):runtime/src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Unix.cs
Lines 33 to 38 in d033e1a
And we use it's internal API surface to workaround this limitation is
CoreLib(mind the call to RandomAccess.WriteAtOffset (internal), not RandomAccess.Write (public):runtime/src/libraries/System.Private.CoreLib/src/System/IO/File.cs
Line 382 in d033e1a
We should relax the public requirements and make
RandomAccesswork with non-seekable files.This is going to require:
Everything motioned above should be a single PR. In the same or separate PR, the Unix implementation of overloads that accept multiple buffers should start using
readvandwritevsys-calls. This should be relatively easy (just search forpreadvandpwritevand reuse the patterns)But it's going to allow to:
FileStreamto a very lightweightRandomAccesseverywhere where we don't need buffering in BCL (and ofc outside of BCL). Sample perf gains: File.*AllText* optimizations #58167 (comment)</issue_description>
<agent_instructions>1. Do exactly what I did in #96711, you can even cherry-pick the git commits if you want.
2. For testing, in
RandomAccess_NonSeekable.GetNamedPipeHandlesAsyncdon't useNamedPipeClientStreamorNamedPipeServerStreamjust use the bran newSafeFileHandle.CreateAnonymousPipe(out readHandle, out writeHandle, bool asyncRead = false, bool asyncWrite = false)API.3. Search the repo for places where we have
SafeFileHandleand could use theRandomAccessAPI instead of performing direct sys-calls. Use the API there.4. Make sure you run all the
src/libraries/System.Runtime/tests/System.IO.FileSystem.Teststest and they are passing before you send the PR.</agent_instructions>Comments on the Issue (you are @copilot in this section)
@ Tagging subscribers to this area: @dotnet/area-system-io See info in [area-owners.md](https://github.com/dotnet/runtime/blob/main/docs/area-owners.md) if you want to be subscribed.Issue Details
In .NET 6 we have introduced new type called
RandomAccessthat allows for reading and writing to specific file offset.As of today, all it's
Read*andWrite*methods throw when given handle points to a non-seekable file like socket or pipe:runtime/src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.cs
Line 251 in d033e1a
But it's not a problem for it's internal implementation (used by
FileStream):runtime/src/libraries/System.Private.CoreLib/src/System/IO/RandomAccess.Unix.cs
Lines 33 to 38 in d033e1a
And we use it's internal API surface to workaround this limitation is
CoreLib(mind the call to RandomAccess.WriteAtOffset (internal), not RandomAccess.Write (public):runtime/src/libraries/System.Private.CoreLib/src/System/IO/File.cs
Line 382 in d033e1a
We should relax the public requirements and make
RandomAccesswork with non-seekable files.This is going to require:
Everything motioned above should be a single PR. In the same or...
🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.