add livekit_agent_simulation service, agent remote session proto & cli dev proto by theomonnom · Pull Request #1404 · livekit/protocol

theomonnom · 2026-02-08T05:44:25Z

No description provided.

changeset-bot · 2026-02-08T05:44:30Z

🦋 Changeset detected

Latest commit: 23813dc

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

💥 An error occurred when fetching the changed packages and changesets in this PR

Some errors occurred when validating the changesets config:
The package or glob expression "github.com/livekit/protocol" specified in the `fixed` option does not match any package in the project. You may have misspelled the package name or provided an invalid glob expression. Note that glob expressions must be defined according to https://www.npmjs.com/package/micromatch.

…ivekit/protocol into theo/agent-simulations-service

davidzhao · 2026-02-18T21:19:12Z

+    STATUS_FAILED = 3;
+  }
+
+  message Job {


do we want to nest this message? seems like it gets complicated to use.. vs SimulationJob

davidzhao · 2026-02-18T21:23:39Z

+  string error = 9;
+  string group_id = 10;
+  // Set as participant attributes on the simulation participant's access token.
+  map<string, string> metadata = 11;


let's be explicit so it's not confused with participant metadata

Suggested change

map<string, string> metadata = 11;

map<string, string> participant_attributes = 11;

davidzhao · 2026-02-18T21:25:19Z

+  string agent_name = 2;
+  string agent_description = 3;
+  int32 num_simulations = 4;
+  string group_id = 5;


how do I just run a single simulation scenario?

davidzhao · 2026-02-18T21:26:52Z

+  int64 created_at = 7;
+  Status status = 8;
+  string error = 9;
+  string group_id = 10;


can you create a scenario without a group? and then assign it to a group later? or should it have a default group?

davidzhao · 2026-02-18T21:28:37Z

+  string instructions = 3;
+  string agent_expectations = 4;
+  string group_id = 5;
+  map<string, string> metadata = 6;


should source_room_id be part of this too? same comment about metadata

davidzhao · 2026-02-18T21:30:09Z

+}
+message DeleteScenarioResponse {}
+
+message UpdateScenarioRequest {


would it be simpler to just include the Scenario message here and also in Create?

Ocupe · 2026-03-10T11:10:09Z

Hey @theomonnom, nice work! I have a suggestion for a logically identical structure, just with more nesting. Do what you like with it, but I find it much easier to reason about and the generated struct names are easier to read.
For example: top with nested messages, below without.

&backend.AccessRecord_Subject_LivekitSupport{}
// vs:
&backend.AccessRecordSubjectLivekitSupport{}

Branch based of yours: https://github.com/livekit/protocol/blob/js/agent-sim/protobufs/livekit_agent_simulation.proto

Like I said, do what you want with it, I just wanted to share this as I found it really valuable.

Ocupe

Looking good. 👍
I think the api that list simulation runs also needs to support filtering and pagination so that we can implement and use it on dashboard.

- Move response types from SessionRequest into SessionResponse - Rename SendMessage/SendMessageResponse to RunInput/RunInputResponse - Add OverlappingSpeech and SessionUsageUpdated events, remove MetricsCollected - Add model usage types (LLM, TTS, STT, Interruption) - Add GetSessionState, GetRTCStats, GetSessionUsage request/response - Restructure simulation proto with nested CRUD pattern - Migrate simulation created_at from int64 to google.protobuf.Timestamp - Add pagination and status filter to ListSimulationRuns

Moved from livekit.agent package to livekit to break the agent -> livekit import cycle.

theomonnom · 2026-03-17T01:28:41Z

Looking good. 👍 I think the api that list simulation runs also needs to support filtering and pagination so that we can implement and use it on dashboard.

Done, using TokenPagination

davidzhao

lg, just a few naming nits

davidzhao · 2026-03-17T04:34:35Z

+  string url = 5;
+  string token = 6;
+  string worker_id = 7;
+  bool fake_job = 8;


nit: fake -> mock

davidzhao · 2026-03-17T04:36:24Z

+  uint32 samples_per_channel = 4;
+}
+
+enum SessionAgentState {


these should be re-usable right? should they just be AgentState & UserState?

also recommend the enums to be abbreviated, i.e.:

AS_INITIALIZING

davidzhao · 2026-03-17T04:39:24Z

+  repeated ModelUsage model_usage = 1;
+}
+
+message SessionEvent {


prefix with AgentSession to disambiguate?

Suggested change

message SessionEvent {

message AgentSessionEvent {

davidzhao · 2026-03-17T04:41:26Z

+message SessionAudioPlaybackClear {}
+message SessionAudioPlaybackFinished {}
+
+message AgentSessionMessage {


so these will be text or data streams, while the SessionRequest/Responses are using RPCs?

They call all use text/data streams because they have a request_id
Also used for our TCP transport

…_job -> mock_job

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

wip

2d58fe5

github-actions bot and others added 3 commits February 8, 2026 05:45

generated protobuf

d08dd97

Merge branch 'main' into theo/agent-simulations-service

e781785

Merge branch 'theo/agent-simulations-service' of https://github.com/l…

5e1c2cf

…ivekit/protocol into theo/agent-simulations-service

davidzhao reviewed Feb 10, 2026

View reviewed changes

Comment thread protobufs/livekit_agent_simulation.proto Outdated

Comment thread protobufs/livekit_agent_simulation.proto Outdated

Comment thread protobufs/livekit_agent_simulation.proto Outdated

theomonnom and others added 2 commits February 10, 2026 15:26

Update livekit_agent_simulation.proto

4a7927b

generated protobuf

52062a4

theomonnom requested a review from davidzhao February 11, 2026 22:38

theomonnom and others added 4 commits February 18, 2026 13:11

add Simulation scenarios (#1417)

0c419ce

generated protobuf

b04fce3

add cancelled

166e02f

generated protobuf

5550b30

davidzhao reviewed Feb 18, 2026

View reviewed changes

Ocupe self-requested a review March 16, 2026 10:52

Ocupe reviewed Mar 16, 2026

View reviewed changes

theomonnom and others added 4 commits March 16, 2026 15:43

generated protobuf

95af52e

add agent dev message proto for CLI hot-reload IPC

0925910

Moved from livekit.agent package to livekit to break the agent -> livekit import cycle.

generated protobuf

6c5fe6d

theomonnom and others added 6 commits March 16, 2026 18:31

renumber session event fields, remove stale comment

9ef7c71

generated protobuf

8315e92

fix agent dev proto comment

460f9d3

generated protobuf

b59d2b1

move agent_dev proto to agent/, replace Job with bytes

8ce5ede

generated protobuf

e08abe1

theomonnom changed the title ~~add livekit_agent_simulation service~~ add livekit_agent_simulation service, agent remote session proto & cli dev proto Mar 17, 2026

davidzhao approved these changes Mar 17, 2026

View reviewed changes

theomonnom and others added 2 commits March 16, 2026 22:52

address review: rename enums, SessionEvent -> AgentSessionEvent, fake…

1997b5b

…_job -> mock_job

generated protobuf

3c60034

This was referenced Mar 17, 2026

use remote session protobuf livekit/agents-playground#197

Merged

replace JSON messages with protobuf messages livekit/agents-js#1139

Merged

theomonnom and others added 7 commits March 17, 2026 12:27

nest AudioFrame and playback messages inside AgentSessionMessage

707ab3c

generated protobuf

8f2d4eb

Create curvy-crews-invent.md

70a9a78

Refine agent session and simulation protos

33c0f74

generated protobuf

10bc09c

generated protobuf

d112790

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

merge main, regenerate protobuf

23813dc

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

theomonnom merged commit 9607166 into main Mar 17, 2026
7 checks passed

theomonnom deleted the theo/agent-simulations-service branch March 17, 2026 20:24

This was referenced Mar 16, 2026

Version Packages #1441

Merged

Version Packages erickirt/protocol#129

Open

	map<string, string> metadata = 11;
	map<string, string> participant_attributes = 11;

Conversation

theomonnom commented Feb 8, 2026

Uh oh!

changeset-bot bot commented Feb 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ocupe commented Mar 10, 2026

Uh oh!

Ocupe left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theomonnom commented Mar 17, 2026

Uh oh!

davidzhao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

changeset-bot bot commented Feb 8, 2026 •

edited

Loading

Ocupe left a comment •

edited

Loading