-
Notifications
You must be signed in to change notification settings - Fork 5.4k
Description
Background and motivation
This is meant to bundle #46663, #27156, and #22838.
There's the usual need of wrapping memory and text-based types in a Stream and given that there are multiple implementations spread across different libraries, it would be ideal to have a standardized API for it directly on .NET.
A few examples of the current options available:
CommunityToolkit.HighPerformance provides AsStream() extension methods for the following:
Memory<byte>
ReadOnlyMemory<byte>
IMemoryOwner<byte>
IBufferWriter<byte>
Nerdbank.Streams also provides AsStream() extensions for the following:
IDuplexPipe
PipeReader/PipeWriter
WebSocket
ReadonlySequence<byte>
IBufferWriter<byte>
While I'm not certain of providing support for all the types listed above, I think a good starting point would be the following based on the evidence shown in the issues I bundled.
API Proposal
Originally proposed by @bartonjs on #82801 (comment).
Static factory methods on System.IO.Stream for CoreLib types, with a C#14 extension type in System.Memory.dll for ReadOnlySequence<byte>.
// In System.Private.CoreLib (System.Runtime.dll)
namespace System.IO
{
public partial class Stream
{
public static Stream FromText(string text, Encoding? encoding = null);
public static Stream FromText(ReadOnlyMemory<char> text, Encoding? encoding = null);
public static Stream FromReadOnlyData(ReadOnlyMemory<byte> data);
public static Stream FromWritableData(Memory<byte> data);
}
}
// In System.Memory.dll (C#14 extension type)
namespace System.IO
{
public static class ReadOnlySequenceStreamExtensions
{
extension(Stream)
{
public static Stream FromReadOnlyData(ReadOnlySequence<byte> sequence);
}
}
}Pros:
- Unified developer experience - all methods appear as
Stream.From*()via IntelliSense, regardless of underlying assembly - Solves layering cleanly - C#14 extensions let System.Memory.dll contribute statics to
Streamwithout modifying CoreLib - Maximum discoverability (FDG: "frameworks must be usable without the need for documentation")
Cons:
- Not all .NET languages support C#14 extension syntax; callers in those languages would use
ReadOnlySequenceStreamExtensions.FromReadOnlyData(...)instead of the projectedStream.FromReadOnlyData(...) - Statics on
Streamin CoreLib are only available to .NET 10+ consumers; multi-targeting libraries (netstandard2.0) cannot use them
API Usage
Example 1:
static HttpClient client = new HttpClient();
static void SendString(string str)
{
var request = new HttpRequestMessage(HttpMethod.Post, "contoso.com");
request.Content = new StreamContent(Stream.FromText(str));
client.Send(request);
}Example 2:
static unsafe void DoStreamOverSpan(ReadOnlySpan<byte> span)
{
fixed (byte* ptr = &MemoryMarshal.GetReference(span))
{
using (MemoryManager<byte> manager = new PointerMemoryManager<byte>(ptr, span.Length))
{
using Stream s = Stream.FromReadOnlyData(manager.Memory);
// Do something with it...
}
}
}Example 3:
// Pipeline protocol deserialization with System.IO.Pipelines
ReadResult result = await pipeReader.ReadAsync();
ReadOnlySequence<byte> buffer = result.Buffer;
// Stream-only deserializer (e.g., XmlSerializer, legacy APIs)
using var stream = Stream.FromReadOnlyData(buffer);
var config = (AppConfig)new XmlSerializer(typeof(AppConfig)).Deserialize(stream);
pipeReader.AdvanceTo(buffer.End);Alternative Design 1
Static factory methods on System.IO.Stream for CoreLib types, with a classic extension method on ReadOnlySequence<byte> in System.Memory.dll.
// In System.Private.CoreLib (System.Runtime.dll)
namespace System.IO
{
public partial class Stream
{
public static Stream FromText(string text, Encoding? encoding = null);
public static Stream FromText(ReadOnlyMemory<char> text, Encoding? encoding = null);
public static Stream FromReadOnlyData(ReadOnlyMemory<byte> data);
public static Stream FromWritableData(Memory<byte> data);
}
}
// In System.Memory.dll (classic extension method)
namespace System.Buffers
{
public static class ReadOnlySequenceExtensions
{
public static Stream AsStream(this ReadOnlySequence<byte> sequence);
}
}Pros:
- Uses only well-established API patterns
- The use of C#14 static extensions in BCL public API is a new pattern still being evaluated by the review board.
AsStream()follows existing precedent (IEnumerable<T>.ToList(),HttpContent.ReadFromJsonAsync()- FDG 5.6)- Works in all .NET languages (no extension member syntax dependency)
Cons:
- Inconsistent API pattern:
Stream.FromReadOnlyData(memory)vssequence.AsStream()for the same conceptual operation AsStream()is less discoverable — requires knowing to look onReadOnlySequence<byte>instead ofStream- In case we want to go all-in with
AsStream()for all types, we cannot extendstring[FDG 5.6]: Besides injecting an unrelated concept into a fundamental type, sincestringis one of the most common types, the pollution in IntelliSense would be massive.
Alternative Design 2
Move ReadOnlySequence<T> to System.Private.CoreLib to enable adding the stream wrapper as a static method directly on Stream, consistent with the other CoreLib types.
From #82801 (comment) and ViveliDuCh#1 (comment)
// In System.Private.CoreLib (System.Runtime.dll)
// Note: This requires moving ReadOnlySequence<T> from System.Memory.dll to System.Private.CoreLib
namespace System.IO
{
public partial class Stream
{
public static Stream FromText(string text, Encoding? encoding = null);
public static Stream FromText(ReadOnlyMemory<char> text, Encoding? encoding = null);
public static Stream FromReadOnlyData(ReadOnlyMemory<byte> data);
public static Stream FromReadOnlyData(ReadOnlySequence<byte> sequence);
public static Stream FromWritableData(Memory<byte> data);
}
}Pros:
- Fully consistent — all methods are real statics on
Stream, single namespace and entry point - Maximum discoverability via IntelliSense (FDG: "frameworks must be usable without the need for documentation")
Cons:
- Requires moving
ReadOnlySequence<T>to CoreLib - too much change for a single API scenario - Breaks
ReadOnlySequence<T>availability on older targets (currently available via NuGet on netstandard2.0)
Alternative Design 3
Place all factory methods in a StreamFactory class in System.Memory.dll (which depends on CoreLib where System.IO.Stream lives).
As suggested by @adamsitnik in this comment.
// In System.Memory.dll
namespace System.IO
{
public static class StreamFactory
{
public static Stream FromText(string text, Encoding? encoding = null);
public static Stream FromText(ReadOnlyMemory<char> text, Encoding? encoding = null);
public static Stream FromReadOnlyData(ReadOnlyMemory<byte> data);
public static Stream FromReadOnlyData(ReadOnlySequence<byte> sequence);
public static Stream FromWritableData(Memory<byte> data);
}
}Pros:
- No layering issues — all methods in one class in System.Memory.dll, which can reference both CoreLib and System.Memory types
- Broader reach potential if System.Memory.dll ships as a NuGet for older frameworks
Cons:
- Lower discoverability — developers look for stream creation on
Stream, not onStreamFactory(FDG: "Factories sacrifice discoverability, usability, and consistency for implementation flexibility") - No IntelliSense guidance when typing
Stream.- users must know to look for a separate class - Generally less preferred when constructors/static methods are viable
Locations
The following dotnet/runtime production code could consume this API:
| Library | File | Usage |
|---|---|---|
| System.Net.Http | ReadOnlyMemoryContent.cs | ✅ Yes |
| System.Private.Xml | XmlPreloadedResolver.cs | ✅ Yes |
| System.Private.DataContractSerialization | JsonXmlDataContract.cs | ✅ Yes |
| System.Memory.Data | BinaryData.cs | ❌ No |
| System.Configuration.ConfigurationManager | ImplicitMachineConfigHost.cs | ❌ No |
| System.Resources.Extensions | DeserializingResourceReader.cs | ❌ No |
Current Limitation: Libraries multi-targeting netstandard2.0/net462 cannot use the CoreLib-based API for older targets.
Notes
-
ReadOnlySequence<byte> wrapper need has evolved: Since #27156 was filed (2018), most major serializers (
System.Text.Json,MessagePack,protobuf-net) now acceptReadOnlySequence<byte>directly. The remaining gap is Stream-only APIs (XmlSerializer, legacy deserializers) and frameworks like HybridCache (#100434) whereStreamremains the most common serializer fallback.PipeReader.AsStream()does not cover this gap —PipeReader.Create(sequence).AsStream()produces a non-seekable stream, while consumers likeXmlSerializerandDataContractSerializeruseLength/Seekfor buffer sizing and format detection. A purpose-built wrapper provides seekable, zero-copy access matching in-memory stream semantics.
-
Partial coverage of #100434: This proposal addresses the
ReadOnlySequence<byte>→Streamportion of that issue. TheIBufferWriter<byte>→Streamwrapper is not included here and would be a separate proposal.
Open questions
- Layering consensus for
Stream.FromvsTo/AsStream: Is the overall consensus, regarding the layering interpretation, thatstring, ReadOnlyMemory<char>, Memory<byte>, and ReadOnlyMemory<byte>getStream.From*(as types more primitive than Stream) and thatReadOnlySequence<byte>getsTo/AsStream(as a type at the same level or higher)?- Naming consistency: Does the
Stream.From*naming present a coherent mental model for stream creation across text, memory, and sequences?
- Naming consistency: Does the
- ReadOnlySequence layering consensus: What's the consensus on the layering discussion for
ReadOnlySequence<byte>? See remaining concerns from review.- Should the
ReadOnlySequence<byte>overload use a C#14 static extension on Stream or a classic extension method in System.Memory.dll? See PR discussion, preliminary review, and subsequent discussion.- If C#14 static extension approach is used, the fallback invocation for non-C#14 languages becomes
ReadOnlySequenceStreamExtensions.FromReadOnlyData(...), does the extension class name need to follow a naming convention that preserves discoverability alongsideStream.FromReadOnlyData(...)? - If classic extension method is preferred, is the naming asymmetry (
Stream.FromReadOnlyData(ROM<byte>)vssequence.AsStream()) acceptable?
- If C#14 static extension approach is used, the fallback invocation for non-C#14 languages becomes
- Should the
- [Based on preliminary api review] Parameter naming: Should the
encodingparameter inFromTextbe renamed tostreamEncodingto disambiguate its purpose? i.e.Stream.FromText(myString, streamEncoding: Encoding.Latin1)- The parameter specifies the encoding used for the bytes in the resulting stream, not the encoding of the source text. - Encoding default (UTF-8 vs UTF-16): Is the
encodingparameter as nullable with a UTF-8 default correct inFromText(string text, Encoding? encoding = null)? - Seekability of
ReadOnlyTextStream: Should the text stream returned byFromTextbe seekable? The current prototype is seekable with the tradeoff. - Exposing custom stream types: Do we want to expose the custom stream types used to back the static factory methods being proposed in this API (e.g.,
ReadOnlyTextStream), or should they remain internal? - General
Stream.FromText()/MemoryStream.FromText(): Is there a need or desire for [1] a generalStream.FromText()onStreamthat returns a stream without specifying the concrete type (not generic), and [2] something likeMemoryStream.FromText()which explicitly returns aMemoryStream, or similar for other derived types? - Should
FromTextexpose an optionalbufferSizeparameter (followingStreamReader's constructor pattern), or keep the encoding buffer size as a fixed internal detail?
Risks
No response