* Realtime Client Proposal

* Replace [Experimental("MEAI001")] with DiagnosticIds.Experiments.AIRealTime

* Apply suggestions from code review

Co-authored-by: Copilot <[email protected]>

* Remove AsIRealtimeClient extension method

The extension method on OpenAIClient was not useful because it
completely ignored the OpenAIClient instance - only validating it
for null before creating a new OpenAIRealtimeClient with the
separately provided apiKey and model parameters.

Users can construct OpenAIRealtimeClient directly instead.

* Address PR review comments

- Fix RealtimeSessionExtensions XML doc to reference IRealtimeSession
  instead of IChatClient
- Replace non-standard <ref name> tags with <see cref> in
  RealtimeServerMessageType.cs for proper IntelliSense/doc rendering
- Fix ResponseDone doc summary to say 'completed' instead of 'created'
- Add missing Throw.IfNull(updates) in LoggingRealtimeSession
  .GetStreamingResponseAsync for consistency with other sessions

* Fix multiple issues found during code review

- Split RealtimeServerMessageType enum: add ResponseOutputItemDone
  and ResponseOutputItemAdded to distinguish per-item events
  (response.output_item.done, conversation.item.done) from
  whole-response events (response.done, response.created)

- Fix function result serialization: use JsonSerializer.Serialize()
  instead of ToString() to properly serialize complex objects

- Fix OTel streaming duration: start stopwatch at method entry
  instead of immediately before recording, so duration histogram
  measures actual streaming time

- URL-encode model name in WebSocket URI for defensive safety

- Fix OTel metadata tag ordering: apply user metadata before
  standard tags so standard OTel attributes take precedence
  if keys collide

* Remove duplicate internal FunctionInvocationStatus enum

* Remove stale CreateToolsMap calls before the loop

* Remove FunctionInvocationResultInternal; use public FunctionInvocationResult directly

* Add IAsyncDisposable to IRealtimeSession; store and await receive task in OpenAIRealtimeSession

* Revert API baseline JSON changes

* Clarify audio/text token counts are subsets of Input/OutputTokenCount

* Replace ToolChoiceMode enum with ChatToolMode for cross-modality consistency

* Unify TranscriptionOptions across Realtime and SpeechToText

- Move TranscriptionOptions from Realtime/ to SpeechToText/ folder
- Change experimental flag from AIRealTime to AISpeechToText
- Make properties nullable with parameterless constructor
- Rename Language to SpeechLanguage, Model to ModelId
- Replace SpeechToTextOptions.ModelId and .SpeechLanguage with Transcription property
- Update all consumers and tests

* Use HasTopLevelMediaType instead of StartsWith for audio check

* Add API compat suppressions for SpeechToTextOptions breaking changes

* Replace string Eagerness with SemanticEagerness struct (ChatRole pattern)

* Remove InternalsVisibleToTest from Microsoft.Extensions.AI.OpenAI

- Make OpenAIRealtimeSession constructor and ConnectAsync public
- Remove InternalsVisibleToTest from csproj
- Remove OpenAIRealtimeSessionSerializationTests (depended on internal ConnectWithWebSocketAsync)

Co-authored-by: Copilot <[email protected]>

* Add RawRepresentationFactory to RealtimeSessionOptions

Add Func<IRealtimeSession, object?>? RawRepresentationFactory property following
the same pattern used by ChatOptions, EmbeddingGenerationOptions, and other
abstraction options types. Add note in OpenAIRealtimeSession to consume the
factory when switching to the OpenAI SDK.

* Remove OpenAI-specific tracing properties from RealtimeSessionOptions

Remove EnableAutoTracing, TracingGroupId, TracingWorkflowName, and
TracingMetadata from the abstraction layer. These are OpenAI-specific
and should be configured via RawRepresentationFactory when the OpenAI
SDK dependency is added.

* Remove AIFunction/HostedMcpServerTool properties, use ChatToolMode

Remove redundant AIFunction and HostedMcpServerTool properties from
RealtimeSessionOptions and RealtimeClientResponseCreateMessage. Callers
should use ChatToolMode.RequireSpecific(functionName) instead.

Update OpenAI serialization to emit structured tool_choice JSON object
when RequireSpecific is used. Update OpenTelemetry and tests accordingly.

* Improve XML docs for Usage on response and transcription messages

* Remove leftover temp files

* Improve ConversationId XML docs on RealtimeServerResponseCreatedMessage

* Split Text/Audio properties on RealtimeServerOutputTextAudioMessage

Add separate Audio property for Base64-encoded audio data. Text is now
only used for text and transcript content. Update OpenAI parser,
OpenTelemetry session, and tests accordingly.

* Replace RealtimeServerMessageType enum with readonly struct

Follow the ChatRole smart-enum pattern: readonly struct with string
Value, IEquatable, operators, and JsonConverter. Providers can now
define custom message types by constructing new instances.

Update pattern-matching in OpenTelemetryRealtimeSession to use ==
comparisons instead of constant patterns.

* Move Parameter into ErrorContent.Details on error messages

Remove the Parameter property from RealtimeServerErrorMessage and
map error.param to ErrorContent.Details instead. Improve ErrorEventId
XML docs to clarify it correlates to the originating client event.

* Add RawRepresentation property to RealtimeContentItem

Add object? RawRepresentation to hold the original provider data
structure, following the same pattern as other types in the
abstraction layer (e.g., ChatMessage). Updated tests accordingly.

* Rename Metadata to AdditionalProperties for consistency

Rename the Metadata property to AdditionalProperties on both
RealtimeClientResponseCreateMessage and RealtimeServerResponseCreatedMessage
to be consistent with the established pattern used across the AI
abstractions (ChatMessage, ChatOptions, AIContent, etc.). Updated
XML docs, OpenAI provider, OTel session, and tests accordingly.

* Improve MaxOutputTokens XML docs to clarify modality scope

Clarify that MaxOutputTokens is a total budget across all output
modalities (text, audio) and tool calls, not per-modality.

* Improve XML docs on RealtimeClientResponseCreateMessage properties

Clarify that ExcludeFromConversation creates an out-of-band response
whose output is not added to conversation history. Document that
Instructions, Tools, ToolMode, OutputModalities, OutputAudioOptions,
and OutputVoice are per-response overrides of session configuration.

* Improve RealtimeClientResponseCreateMessage class-level XML doc

Clarify that this message triggers model inference and that its
properties are per-response overrides of session configuration.

* Rename EventId to MessageId for terminology consistency

Rename EventId to MessageId on RealtimeClientMessage and
RealtimeServerMessage, and ErrorEventId to ErrorMessageId on
RealtimeServerErrorMessage. The abstraction uses 'message' terminology
throughout (class names, docs, method signatures), so properties
should match. The OpenAI provider maps MessageId to/from the wire
protocol's event_id field.

* Remove blank line between using directives in audio buffer messages

* Rename RealtimeAudioFormat.Type to MediaType for consistency

Rename to match the established MediaType naming convention used
across the abstractions (DataContent, HostedFileContent, UriContent,
ImageGenerationOptions). Updated OpenAI provider and tests.

* Refactor IRealtimeSession: rename InjectClientMessageAsync to SendClientMessageAsync and remove updates parameter from GetStreamingResponseAsync

- Rename InjectClientMessageAsync -> SendClientMessageAsync across all implementations
- Remove IAsyncEnumerable<RealtimeClientMessage> updates parameter from GetStreamingResponseAsync
- Move per-message telemetry from WrapClientMessagesForTelemetryAsync into SendClientMessageAsync override in OpenTelemetryRealtimeSession
- Delete WrapUpdatesWithLoggingAsync from LoggingRealtimeSession
- Delete WrapClientMessagesForTelemetryAsync from OpenTelemetryRealtimeSession
- Update AnonymousDelegatingRealtimeSession delegate signature
- Update RealtimeSessionBuilder.Use overload signature
- Update all tests to use new API

Co-authored-by: Copilot <[email protected]>

* Make RealtimeSessionOptions and related types immutable with init-only properties and IReadOnlyList

* Fix XML doc param order, add null validation to constructors

* OpenAI Realtime Provider using OpenAI SDK

* Fix typo: add missing space in comment ('IDsince' -> 'ID since')

* Make CreateSessionAsync return non-nullable IRealtimeSession

- Change IRealtimeClient.CreateSessionAsync return type from Task<IRealtimeSession?> to Task<IRealtimeSession>
- Remove exception-swallowing catch blocks in OpenAIRealtimeClient.CreateSessionAsync and OpenAIRealtimeSession.ConnectAsync; let exceptions propagate to callers
- Change ConnectAsync from Task<bool> to Task
- Remove unused System.IO and System.Net.WebSockets usings
- Update tests to expect OperationCanceledException instead of null/false returns

* Remove SemanticEagerness from Abstractions; move eagerness to AdditionalProperties

- Delete SemanticEagerness.cs (OpenAI-specific concept)
- Remove Eagerness property from SemanticVoiceActivityDetection
- Add AdditionalProperties to base VoiceActivityDetection for provider-specific settings
- Update OpenAI provider to read/write eagerness via AdditionalProperties["eagerness"]

* Include CompatibilitySuppressions for SpeechToTextOptions API changes

* Remove MCP-specific types from abstractions; add provider contract docs

* Remove trailing blank line in RealtimeServerErrorMessage

* Clarify ErrorMessageId XML doc to distinguish from base MessageId

* Rename ErrorMessageId to OriginatingMessageId for clarity

* Change ExcludeFromConversation from bool to bool? for provider-default consistency

* Remove blank lines between Experimental attribute and class declaration

* Add null validation to Content property setter

* Make RealtimeAudioFormat.SampleRate non-nullable and remove misleading doc

* Rename IRealtimeSession to IRealtimeClientSession and SendClientMessageAsync to SendAsync

* Fix null DefaultConversationConfiguration when ExcludeFromConversation is unset

* Remove IDisposable from IRealtimeClientSession, keep only IAsyncDisposable

* Remove ConversationId from RealtimeServerResponseCreatedMessage

* Rename ResponseCreateMessage and ConversationItemCreateMessage to CreateResponseMessage and CreateConversationItemMessage

* Rename RealtimeContentItem to RealtimeConversationItem

* Remove VoiceSpeed from RealtimeSessionOptions and use RawRepresentationFactory seed pattern

* Remove UpdateAsync from IRealtimeClientSession, use message-based approach

Replace UpdateAsync method on IRealtimeClientSession with a new
RealtimeClientSessionUpdateMessage type sent via SendAsync. This avoids
requiring all providers to implement session updates (e.g. Gemini does
not support mid-session updates).

- Add RealtimeClientSessionUpdateMessage with Options property
- Remove UpdateAsync from IRealtimeClientSession and DelegatingRealtimeSession
- Move update logic to OpenAIRealtimeSession.SendAsync handler
- Remove UpdateAsync overrides from Logging and OpenTelemetry sessions
- Update all tests to use SendAsync with the new message type

* Rename realtime session classes to include Client in name

Align implementation class names with IRealtimeClientSession interface
naming. All session-related classes now include 'Client':

- DelegatingRealtimeSession -> DelegatingRealtimeClientSession
- OpenAIRealtimeSession -> OpenAIRealtimeClientSession
- LoggingRealtimeSession -> LoggingRealtimeClientSession
- OpenTelemetryRealtimeSession -> OpenTelemetryRealtimeClientSession
- FunctionInvokingRealtimeSession -> FunctionInvokingRealtimeClientSession
- RealtimeSessionBuilder -> RealtimeClientSessionBuilder
- RealtimeSessionExtensions -> RealtimeClientSessionExtensions
- TestRealtimeSession -> TestRealtimeClientSession
- All related builder extension and test classes updated

* Simplify DisposeAsyncCore to return ValueTask directly

Remove unnecessary async/await in DelegatingRealtimeClientSession.DisposeAsyncCore,
returning InnerSession.DisposeAsync() directly instead.

* Convert RealtimeSessionKind from enum to extensible string struct

Convert RealtimeSessionKind from a closed enum to a readonly struct
following the same pattern as RealtimeServerMessageType and
ChatFinishReason. This allows providers to define custom session kinds.

Rename Realtime -> Conversation for clarity, avoiding the redundant
RealtimeSessionKind.Realtime naming.

* Remove NoiseReductionOptions from abstraction

NoiseReductionOptions is OpenAI-specific and not supported by other
providers. Remove it from the abstraction layer; users who need noise
reduction can use provider-specific options.

* Remove VoiceActivityDetection hierarchy from abstraction

VoiceActivityDetection, ServerVoiceActivityDetection, and
SemanticVoiceActivityDetection are too OpenAI-specific. Remove them
from the abstraction; users who need VAD can configure it through
provider-specific options via RawRepresentationFactory.

* Remove AnonymousDelegatingRealtimeClientSession and anonymous Use overload

Remove the anonymous Use overload that only intercepts
GetStreamingResponseAsync without the ability to intercept SendAsync,
making it too limited to be useful in practice.

* Fix JSON injection in MessageId by using JsonSerializer.Serialize

* Remove meaningless duration metric from SendAsync in OTel session

The stopwatch was started and recorded within the same block without
wrapping the actual SendAsync call, measuring nothing useful. Duration
metrics remain in GetStreamingResponseAsync where they measure actual
streaming time.

* Refactor builder to operate on IRealtimeClient instead of IRealtimeClientSession

- Create DelegatingRealtimeClient base class (replaces DelegatingRealtimeClientSession)
- Create RealtimeClientBuilder (replaces RealtimeClientSessionBuilder)
- Create FunctionInvokingRealtimeClient, LoggingRealtimeClient, OpenTelemetryRealtimeClient
- Make session middleware types (FunctionInvokingRealtimeClientSession, etc.) internal
- Rename builder extension classes to remove 'Session' from names
- Add RealtimeClientExtensions and RealtimeClientBuilderRealtimeClientExtensions
- Use DiagnosticIds.Experiments.AIRealTime for Experimental attributes
- Update all tests to use public client APIs only

* Rename message classes: move RealtimeClient/RealtimeServer before Message

Rename pattern: RealtimeClient{Name}Message -> {Name}RealtimeClientMessage
Rename pattern: RealtimeServer{Name}Message -> {Name}RealtimeServerMessage

Base classes RealtimeClientMessage and RealtimeServerMessage unchanged.

* Fix cancellation tests to also accept WebSocketException on net462

* Remove PreviousId from CreateConversationItemRealtimeClientMessage as OpenAI-specific

* Document that ResponseId may be null and response lifecycle events may be synthesized

* Document that CreateResponseRealtimeClientMessage may be a no-op for VAD-driven providers

* Add RealtimeResponseStatus constants and document interruption/barge-in via Status property

* Remove session parameter from RealtimeSessionOptions.RawRepresentationFactory

* Fix AOT compatibility errors in OpenAIRealtimeClientSession by using source-generated JSON serialization

* Fix SpeechToTextOptionsTests for unified TranscriptionOptions after upstream merge

* Revert SpeechToTextOptions changes per review feedback

Restore ModelId and SpeechLanguage as direct properties on
SpeechToTextOptions instead of nesting them under a Transcription
property. TranscriptionOptions class remains for RealtimeSessionOptions.

- Restore SpeechToTextOptions.ModelId and .SpeechLanguage properties
- Remove SpeechToTextOptions.Transcription property
- Update all consumers to use direct property access
- Fix SpeechToTextClientMetadata doc reference
- Regenerate CompatibilitySuppressions.xml
- Update all related tests

* Use JsonInclude/Core pattern for experimental UsageDetails properties

Apply the [JsonIgnore] public + [JsonInclude] internal Core backing
property pattern to the 4 experimental audio/text token properties
on UsageDetails, matching the convention used elsewhere (e.g.
ChatOptions.AllowBackgroundResponses, ChatResponse.ContinuationToken).

This enables JSON serialization of these properties when using the
library's JsonSerializerOptions while keeping the public API gated
behind [Experimental].

- InputAudioTokenCount -> InputAudioTokenCountCore
- InputTextTokenCount -> InputTextTokenCountCore
- OutputAudioTokenCount -> OutputAudioTokenCountCore
- OutputTextTokenCount -> OutputTextTokenCountCore
- Add tests for property roundtrip, Add summation, and JSON
  serialization/deserialization of the new properties

* Fix SendAsync to propagate exceptions instead of swallowing them

Replace silent return on cancellation with ThrowIfCancellationRequested,
throw InvalidOperationException when session is not connected, and remove
catch block that swallowed OperationCanceledException, ObjectDisposedException,
and WebSocketException.

Update tests to verify the new exception behavior.

* Replace toolMap dictionary with FindTool delegate pattern

Refactor FunctionInvocationProcessor to accept a Func<string, AITool?>
delegate instead of Dictionary<string, AITool>? to align with main
branch's FindTool pattern and avoid reintroducing the removed toolMap.

* Use IEnumerable<AITool> in FindTool/HasAnyTools to avoid unsafe IList cast

Options.Tools is IReadOnlyList<AITool> which does not implement
IList<AITool>, so the 'as IList<AITool>' cast could silently return
null and ignore tools. Use IEnumerable<AITool> which both IList<T>
and IReadOnlyList<T> implement.

* Document that function invocation blocks message processing loop

Add known limitation note to FunctionInvokingRealtimeClientSession
XML docs explaining that incoming server messages (including user
interruptions) are buffered during function invocation.

* Add distinct ConversationItem message types to prevent double invocation

ConversationItemAdded and ConversationItemDone were incorrectly mapped
to ResponseOutputItemAdded/Done, which could cause the function
invoking session to invoke the same function call twice.

* Remove custom ToolChoice telemetry attribute

Remove gen_ai.request.tool_choice custom attribute that is not part
of the OpenTelemetry GenAI semantic conventions.

* Add null validation to Item property setter

Consistent with constructor which already validates via Throw.IfNull.

* Add <exception> XML doc tags to realtime types

Document ArgumentNullException on constructors and property setters
that validate via Throw.IfNull/Throw.IfNullOrWhitespace.

* Use OrdinalIgnoreCase comparer for MCP headers dictionary

HTTP headers are case-insensitive per RFC 7230.

* Change RealtimeResponseStatus from const fields to static properties

Avoids baking values into consuming assemblies at compile time.

* Add GetRequiredService extension methods for realtime types

Add GetRequiredService and GetRequiredService<T> to both
IRealtimeClient and IRealtimeClientSession extensions, matching
the pattern from IChatClient and IEmbeddingGenerator.

* Remove unused System.Threading.Channels package reference

* Update OTel semantic conventions version reference from v1.39 to v1.40

---------

Co-authored-by: Tarek Mahmoud Sayed <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>