Skip to content

[API Proposal]: Add Stream wrappers for memory and text-based types #82801

@jozkee

Description

@jozkee

Background and motivation

This is meant to bundle #46663, #27156, and #22838.

There's the usual need of wrapping memory and text-based types in a Stream and given that there are multiple implementations spread across different libraries, it would be ideal to have a standardized API for it directly on .NET.

A few examples of the current options available:
CommunityToolkit.HighPerformance provides AsStream() extension methods for the following:
Memory<byte>
ReadOnlyMemory<byte>
IMemoryOwner<byte>
IBufferWriter<byte>

Nerdbank.Streams also provides AsStream() extensions for the following:
IDuplexPipe
PipeReader/PipeWriter
WebSocket
ReadonlySequence<byte>
IBufferWriter<byte>

While I'm not certain of providing support for all the types listed above, I think a good starting point would be the following based on the evidence shown in the issues I bundled.

API Proposal

Originally proposed by @bartonjs on #82801 (comment).
Static factory methods on System.IO.Stream for CoreLib types, with a C#14 extension type in System.Memory.dll for ReadOnlySequence<byte>.

// In System.Private.CoreLib (System.Runtime.dll)
namespace System.IO
{
    public partial class Stream
    {
        public static Stream FromText(string text, Encoding? encoding = null);
        public static Stream FromText(ReadOnlyMemory<char> text, Encoding? encoding = null);

        public static Stream FromReadOnlyData(ReadOnlyMemory<byte> data);
        public static Stream FromWritableData(Memory<byte> data);
    }
}

// In System.Memory.dll (C#14 extension type)
namespace System.IO
{
    public static class ReadOnlySequenceStreamExtensions
    {
        extension(Stream)
        {
            public static Stream FromReadOnlyData(ReadOnlySequence<byte> sequence);
        }
    }
}

Pros:

  • Unified developer experience - all methods appear as Stream.From*() via IntelliSense, regardless of underlying assembly
  • Solves layering cleanly - C#14 extensions let System.Memory.dll contribute statics to Stream without modifying CoreLib
  • Maximum discoverability (FDG: "frameworks must be usable without the need for documentation")

Cons:

  • Not all .NET languages support C#14 extension syntax; callers in those languages would use ReadOnlySequenceStreamExtensions.FromReadOnlyData(...) instead of the projected Stream.FromReadOnlyData(...)
  • Statics on Stream in CoreLib are only available to .NET 10+ consumers; multi-targeting libraries (netstandard2.0) cannot use them

API Usage

Example 1:

static HttpClient client = new HttpClient();
static void SendString(string str)
{
    var request = new HttpRequestMessage(HttpMethod.Post, "contoso.com");
    request.Content = new StreamContent(Stream.FromText(str));
    client.Send(request);
}

Example 2:

static unsafe void DoStreamOverSpan(ReadOnlySpan<byte> span)
{
    fixed (byte* ptr = &MemoryMarshal.GetReference(span))
    {
        using (MemoryManager<byte> manager = new PointerMemoryManager<byte>(ptr, span.Length))
        {
            using Stream s = Stream.FromReadOnlyData(manager.Memory);
            // Do something with it...
        }
    }
}

Example 3:

// Pipeline protocol deserialization with System.IO.Pipelines
ReadResult result = await pipeReader.ReadAsync();
ReadOnlySequence<byte> buffer = result.Buffer;

// Stream-only deserializer (e.g., XmlSerializer, legacy APIs)
using var stream = Stream.FromReadOnlyData(buffer);
var config = (AppConfig)new XmlSerializer(typeof(AppConfig)).Deserialize(stream);

pipeReader.AdvanceTo(buffer.End);

Alternative Design 1

Static factory methods on System.IO.Stream for CoreLib types, with a classic extension method on ReadOnlySequence<byte> in System.Memory.dll.

// In System.Private.CoreLib (System.Runtime.dll)
namespace System.IO
{
    public partial class Stream
    {
        public static Stream FromText(string text, Encoding? encoding = null);
        public static Stream FromText(ReadOnlyMemory<char> text, Encoding? encoding = null);

        public static Stream FromReadOnlyData(ReadOnlyMemory<byte> data);
        public static Stream FromWritableData(Memory<byte> data);
    }
}

// In System.Memory.dll (classic extension method)
namespace System.Buffers
{
    public static class ReadOnlySequenceExtensions
    {
        public static Stream AsStream(this ReadOnlySequence<byte> sequence);
    }
}

Pros:

  • Uses only well-established API patterns
    • The use of C#14 static extensions in BCL public API is a new pattern still being evaluated by the review board.
  • AsStream() follows existing precedent (IEnumerable<T>.ToList(), HttpContent.ReadFromJsonAsync() - FDG 5.6)
  • Works in all .NET languages (no extension member syntax dependency)

Cons:

  • Inconsistent API pattern: Stream.FromReadOnlyData(memory) vs sequence.AsStream() for the same conceptual operation
  • AsStream() is less discoverable — requires knowing to look on ReadOnlySequence<byte> instead of Stream
  • In case we want to go all-in with AsStream() for all types, we cannot extend string [FDG 5.6]: Besides injecting an unrelated concept into a fundamental type, since string is one of the most common types, the pollution in IntelliSense would be massive.

Alternative Design 2

Move ReadOnlySequence<T> to System.Private.CoreLib to enable adding the stream wrapper as a static method directly on Stream, consistent with the other CoreLib types.

From #82801 (comment) and ViveliDuCh#1 (comment)

// In System.Private.CoreLib (System.Runtime.dll)
// Note: This requires moving ReadOnlySequence<T> from System.Memory.dll to System.Private.CoreLib
namespace System.IO
{
    public partial class Stream
    {
        public static Stream FromText(string text, Encoding? encoding = null);
        public static Stream FromText(ReadOnlyMemory<char> text, Encoding? encoding = null);

        public static Stream FromReadOnlyData(ReadOnlyMemory<byte> data);
        public static Stream FromReadOnlyData(ReadOnlySequence<byte> sequence);
        public static Stream FromWritableData(Memory<byte> data);
    }
}

Pros:

  • Fully consistent — all methods are real statics on Stream, single namespace and entry point
  • Maximum discoverability via IntelliSense (FDG: "frameworks must be usable without the need for documentation")

Cons:

  • Requires moving ReadOnlySequence<T> to CoreLib - too much change for a single API scenario
  • Breaks ReadOnlySequence<T> availability on older targets (currently available via NuGet on netstandard2.0)

Alternative Design 3

Place all factory methods in a StreamFactory class in System.Memory.dll (which depends on CoreLib where System.IO.Stream lives).

As suggested by @adamsitnik in this comment.

// In System.Memory.dll
namespace System.IO
{
    public static class StreamFactory
    {
        public static Stream FromText(string text, Encoding? encoding = null);
        public static Stream FromText(ReadOnlyMemory<char> text, Encoding? encoding = null);

        public static Stream FromReadOnlyData(ReadOnlyMemory<byte> data);
        public static Stream FromReadOnlyData(ReadOnlySequence<byte> sequence);

        public static Stream FromWritableData(Memory<byte> data);
    }
}

Pros:

  • No layering issues — all methods in one class in System.Memory.dll, which can reference both CoreLib and System.Memory types
  • Broader reach potential if System.Memory.dll ships as a NuGet for older frameworks

Cons:

  • Lower discoverability — developers look for stream creation on Stream, not on StreamFactory (FDG: "Factories sacrifice discoverability, usability, and consistency for implementation flexibility")
  • No IntelliSense guidance when typing Stream. - users must know to look for a separate class
  • Generally less preferred when constructors/static methods are viable

Locations

The following dotnet/runtime production code could consume this API:

Library File Usage
System.Net.Http ReadOnlyMemoryContent.cs ✅ Yes
System.Private.Xml XmlPreloadedResolver.cs ✅ Yes
System.Private.DataContractSerialization JsonXmlDataContract.cs ✅ Yes
System.Memory.Data BinaryData.cs ❌ No
System.Configuration.ConfigurationManager ImplicitMachineConfigHost.cs ❌ No
System.Resources.Extensions DeserializingResourceReader.cs ❌ No

Current Limitation: Libraries multi-targeting netstandard2.0/net462 cannot use the CoreLib-based API for older targets.

Notes

  • ReadOnlySequence<byte> wrapper need has evolved: Since #27156 was filed (2018), most major serializers (System.Text.Json, MessagePack, protobuf-net) now accept ReadOnlySequence<byte> directly. The remaining gap is Stream-only APIs (XmlSerializer, legacy deserializers) and frameworks like HybridCache (#100434) where Stream remains the most common serializer fallback.

    • PipeReader.AsStream() does not cover this gap — PipeReader.Create(sequence).AsStream() produces a non-seekable stream, while consumers like XmlSerializer and DataContractSerializer use Length/Seek for buffer sizing and format detection. A purpose-built wrapper provides seekable, zero-copy access matching in-memory stream semantics.
  • Partial coverage of #100434: This proposal addresses the ReadOnlySequence<byte>Stream portion of that issue. The IBufferWriter<byte>Stream wrapper is not included here and would be a separate proposal.

Open questions

  1. Layering consensus for Stream.From vs To/AsStream: Is the overall consensus, regarding the layering interpretation, that string, ReadOnlyMemory<char>, Memory<byte>, and ReadOnlyMemory<byte> get Stream.From* (as types more primitive than Stream) and that ReadOnlySequence<byte> gets To/AsStream (as a type at the same level or higher)?
    • Naming consistency: Does the Stream.From* naming present a coherent mental model for stream creation across text, memory, and sequences?
  2. ReadOnlySequence layering consensus: What's the consensus on the layering discussion for ReadOnlySequence<byte>? See remaining concerns from review.
    • Should the ReadOnlySequence<byte> overload use a C#14 static extension on Stream or a classic extension method in System.Memory.dll? See PR discussion, preliminary review, and subsequent discussion.
      • If C#14 static extension approach is used, the fallback invocation for non-C#14 languages becomes ReadOnlySequenceStreamExtensions.FromReadOnlyData(...), does the extension class name need to follow a naming convention that preserves discoverability alongside Stream.FromReadOnlyData(...)?
      • If classic extension method is preferred, is the naming asymmetry (Stream.FromReadOnlyData(ROM<byte>) vs sequence.AsStream()) acceptable?
  3. [Based on preliminary api review] Parameter naming: Should the encoding parameter in FromText be renamed to streamEncoding to disambiguate its purpose? i.e. Stream.FromText(myString, streamEncoding: Encoding.Latin1) - The parameter specifies the encoding used for the bytes in the resulting stream, not the encoding of the source text.
  4. Encoding default (UTF-8 vs UTF-16): Is the encoding parameter as nullable with a UTF-8 default correct in FromText(string text, Encoding? encoding = null)?
  5. Seekability of ReadOnlyTextStream: Should the text stream returned by FromText be seekable? The current prototype is seekable with the tradeoff.
  6. Exposing custom stream types: Do we want to expose the custom stream types used to back the static factory methods being proposed in this API (e.g., ReadOnlyTextStream), or should they remain internal?
  7. General Stream.FromText() / MemoryStream.FromText(): Is there a need or desire for [1] a general Stream.FromText() on Stream that returns a stream without specifying the concrete type (not generic), and [2] something like MemoryStream.FromText() which explicitly returns a MemoryStream, or similar for other derived types?
  8. Should FromText expose an optional bufferSize parameter (following StreamReader's constructor pattern), or keep the encoding buffer size as a fixed internal detail?

Risks

No response

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions