Skip to content

feat(runtime-core): Refactor run agent block, and extract some implementation into the CLI, so that runtime can be reused in vscode extension#342

Draft
tkislan wants to merge 25 commits intomainfrom
tk/agent-vscode-abstract
Draft

feat(runtime-core): Refactor run agent block, and extract some implementation into the CLI, so that runtime can be reused in vscode extension#342
tkislan wants to merge 25 commits intomainfrom
tk/agent-vscode-abstract

Conversation

@tkislan
Copy link
Contributor

@tkislan tkislan commented Mar 13, 2026

Summary by CodeRabbit

  • New Features

    • Agents can insert and execute code blocks and add markdown during notebook runs; inserted blocks are executed and streamed as outputs.
  • Improvements

    • Agent event callbacks can be asynchronous (handlers may await); stream processing awaits callback completion.
    • Agent flows validate API key presence and fail with clearer errors when missing.
    • Agent results now expose only the final output (internal block metadata hidden).
  • Tests

    • Agent tests stub the OpenAI API key, cover success and failure of inserted blocks, and clean up environment.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 13, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 1ceb2056-a243-44aa-b4d9-c694697c85dd

📥 Commits

Reviewing files that changed from the base of the PR and between b6bc0e7 and da7b1d0.

📒 Files selected for processing (1)
  • packages/runtime-core/src/execution-engine.test.ts

📝 Walkthrough

Walkthrough

Agent block execution was refactored to use a context-driven API: agents now receive an AgentBlockContext with openAiToken, mcpServers, notebookContext, and async handlers addAndExecuteCodeBlock and addMarkdownBlock. New result types AddAndExecuteCodeBlockResult and AddMarkdownBlockResult were added. Helpers serializeNotebookContext and createBlocksWithAttachedOutputsFromCollectedOutputs were introduced and exported. AgentBlockResult now returns only finalOutput. onAgentEvent signatures were changed to allow async handlers and the stream processing awaits onAgentEvent. ExecutionEngine and tests adjusted to drive agent-added block insertion/execution via the new context. The CLI run callback was made async when machine output is disabled.

Sequence Diagram

sequenceDiagram
    participant Exec as ExecutionEngine
    participant Agent as Agent (LLM)
    participant Tools as AgentTools
    participant Kernel as Kernel/Runner
    participant Host as Host (onAgentEvent)

    Exec->>Exec: build AgentBlockContext\n(openAiToken, mcpServers, notebookContext, handlers)
    Exec->>Agent: executeAgentBlock(context)
    activate Agent
    Agent->>Tools: addAndExecuteCodeBlock(code) / addMarkdownBlock(content)
    activate Tools
    Tools->>Kernel: execute code / render markdown
    Kernel-->>Tools: outputs & execution metadata
    Tools-->>Exec: AddAndExecuteCodeBlockResult / AddMarkdownBlockResult
    deactivate Tools
    Agent->>Host: emit AgentStreamEvent
    Host-->>Agent: (async) acknowledge (onAgentEvent awaited)
    Agent->>Exec: finish with finalOutput
    deactivate Agent
    Exec->>Exec: collect outputs / serializeNotebookContext, return finalOutput
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Updates Docs ⚠️ Warning PR introduces breaking API changes to packages/runtime-core without updating documentation in README.md, docs/, or AGENTS.md. Update packages/runtime-core/README.md and relevant docs to document refactored API, new types, and async callback signatures for consumers.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed Title accurately describes the main refactoring: extracting agent block implementation to make runtime reusable in VSCode extension.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@codecov
Copy link

codecov bot commented Mar 13, 2026

Codecov Report

❌ Patch coverage is 87.27273% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.21%. Comparing base (ab97044) to head (da7b1d0).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
packages/runtime-core/src/agent-handler.ts 78.26% 5 Missing ⚠️
packages/cli/src/commands/run.ts 0.00% 1 Missing ⚠️
packages/runtime-core/src/execution-engine.ts 96.77% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #342      +/-   ##
==========================================
+ Coverage   82.70%   83.21%   +0.51%     
==========================================
  Files         126      126              
  Lines        7614     7621       +7     
  Branches     2117     2118       +1     
==========================================
+ Hits         6297     6342      +45     
+ Misses       1316     1278      -38     
  Partials        1        1              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/runtime-core/src/execution-engine.test.ts (1)

833-847: 🛠️ Refactor suggestion | 🟠 Major

Cover the new agent helper path with one real runtime test.

Because executeAgentBlock is mocked at the module boundary, these cases never exercise AgentBlockContext.addAndExecuteCodeBlock, AgentBlockContext.addMarkdownBlock, or the async onAgentEvent flow added by this refactor. A regression in inserted-block reporting would still pass here.

As per coding guidelines, "Create comprehensive tests for all new features using Vitest" and "Test edge cases, error handling, and special characters in test files".

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/runtime-core/src/execution-engine.test.ts` around lines 833 - 847,
Add one real runtime test that does not mock executeAgentBlock at the module
boundary so the new helper path is exercised: undo or avoid the module mock for
executeAgentBlock (remove or restore mockExecuteAgentBlock via
vi.unmock/vi.importActual) and call the actual execution flow with the
AGENT_FIXTURE so AgentBlockContext.addAndExecuteCodeBlock,
AgentBlockContext.addMarkdownBlock, and the async onAgentEvent flow run; then
assert that inserted block reporting (addedBlockIds/blockOutputs/finalOutput and
any events) matches expected values. Ensure the test triggers error/edge cases
and special characters from the fixture so the real helper logic and event
handling are covered rather than returning a mocked result.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/runtime-core/src/agent-handler.ts`:
- Around line 18-26: The AgentBlockContext.onAgentEvent type currently forces
Promise<void> which prevents sync handlers and leaves async errors unobserved;
change the onAgentEvent signature to accept void | Promise<void> in the
AgentBlockContext interface and then update the stream processing loop to await
the result of each call to onAgentEvent for events like "text_delta",
"reasoning_delta", "tool_called", and "tool_output" so both sync and async
handlers are supported and errors are propagated; also update the corresponding
ExecutionOptions type in execution-engine.ts to match the new void |
Promise<void> signature so callers compile.

In `@packages/runtime-core/src/execution-engine.ts`:
- Around line 249-287: addAndExecuteCodeBlock currently only records
outputs/executionCount and loses success/error details; change it to capture and
persist the full execution result from kernel.execute (including success,
outputs, executionCount, and any error or computed outputText) into blockOutputs
and collectedOutputs so that a failing result (result.success === false) is not
treated as success and thrown exceptions are converted to a unified result
object and passed to the same flow; specifically update the logic around
kernel.execute, the construction pushed to blockOutputs and collectedOutputs,
and the returned value so both tool responses and host callbacks (e.g.,
onBlockDone) receive the identical full result object rather than just
outputs/executionCount.

---

Outside diff comments:
In `@packages/runtime-core/src/execution-engine.test.ts`:
- Around line 833-847: Add one real runtime test that does not mock
executeAgentBlock at the module boundary so the new helper path is exercised:
undo or avoid the module mock for executeAgentBlock (remove or restore
mockExecuteAgentBlock via vi.unmock/vi.importActual) and call the actual
execution flow with the AGENT_FIXTURE so
AgentBlockContext.addAndExecuteCodeBlock, AgentBlockContext.addMarkdownBlock,
and the async onAgentEvent flow run; then assert that inserted block reporting
(addedBlockIds/blockOutputs/finalOutput and any events) matches expected values.
Ensure the test triggers error/edge cases and special characters from the
fixture so the real helper logic and event handling are covered rather than
returning a mocked result.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 5a7edfc5-3620-40b0-95cd-66d1fc17f014

📥 Commits

Reviewing files that changed from the base of the PR and between 23d8b03 and dca6b0c.

📒 Files selected for processing (4)
  • packages/cli/src/commands/run.ts
  • packages/runtime-core/src/agent-handler.ts
  • packages/runtime-core/src/execution-engine.test.ts
  • packages/runtime-core/src/execution-engine.ts

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/runtime-core/src/agent-handler.ts (1)

237-239: 🧹 Nitpick | 🔵 Trivial

Consider explicit undefined handling.

JSON.stringify(undefined) returns undefined. Current nullish coalescing works, but explicit check is clearer.

💡 Optional clarity improvement
-        const outputStr = typeof toolOutput === 'string' ? toolOutput : (JSON.stringify(toolOutput) ?? '')
+        const outputStr = toolOutput === undefined ? '' : typeof toolOutput === 'string' ? toolOutput : JSON.stringify(toolOutput)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/runtime-core/src/agent-handler.ts` around lines 237 - 239, The code
that builds outputStr in agent-handler.ts should explicitly handle undefined
instead of relying on JSON.stringify(undefined) returning undefined; update the
logic in the block around toolOutput / outputStr (variables toolOutput and
outputStr, and the context.onAgentEvent call) so that if toolOutput is undefined
you set outputStr to '' (empty string), otherwise if it's a string keep it, and
otherwise set it to JSON.stringify(toolOutput); ensure the resulting outputStr
is always a string before calling context.onAgentEvent with type 'tool_output'.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/runtime-core/src/agent-handler.ts`:
- Line 104: Remove the leftover commented-out line "const outputs =
collectedOutputs.get(block.id)" in the agent handler; locate the commented
statement near the logic that references collectedOutputs and block.id (in the
function handling agent blocks) and delete it so no dead commented code remains,
ensuring there are no other unused commented variables left after the refactor.
- Around line 69-76: The map callback in blocks.map is shadowing and redundantly
re-reading collectedOutputs.get(block.id): remove the inner const outputs =
collectedOutputs.get(block.id) and the nested if, use the single outputs
variable obtained once (const outputs = collectedOutputs.get(block.id)) and,
when outputs is non-null and outputs.outputs.length > 0, return { ...block,
outputs: outputs.outputs }; otherwise return the original block — update the
callback in agent-handler.ts accordingly to eliminate the unreachable shadowed
declaration.

---

Outside diff comments:
In `@packages/runtime-core/src/agent-handler.ts`:
- Around line 237-239: The code that builds outputStr in agent-handler.ts should
explicitly handle undefined instead of relying on JSON.stringify(undefined)
returning undefined; update the logic in the block around toolOutput / outputStr
(variables toolOutput and outputStr, and the context.onAgentEvent call) so that
if toolOutput is undefined you set outputStr to '' (empty string), otherwise if
it's a string keep it, and otherwise set it to JSON.stringify(toolOutput);
ensure the resulting outputStr is always a string before calling
context.onAgentEvent with type 'tool_output'.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 7e67686a-1083-41b3-bf9a-a0ba5b6b0e4e

📥 Commits

Reviewing files that changed from the base of the PR and between dca6b0c and f3c19fb.

📒 Files selected for processing (2)
  • packages/runtime-core/src/agent-handler.ts
  • packages/runtime-core/src/index.ts

…ution and ensure proper handling of event callbacks
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/runtime-core/src/agent-handler.ts (1)

171-203: ⚠️ Potential issue | 🟠 Major

Cover MCP startup with the same cleanup path.

createMCPClient(...) and client.tools() still run before the current try/finally. If one server fails after earlier clients started, those stdio transports stay open. Track clients incrementally inside the cleanup scope.

Also applies to: 222-252

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/runtime-core/src/agent-handler.ts` around lines 171 - 203, The MCP
clients and their tool enumeration are created outside the try/finally, so a
failure partway leaves open stdio transports; move the whole client startup and
tools collection into the protected scope and track started clients
incrementally (e.g., push each created client from
mergedMcpConfig.map/createMCPClient into a local startedClients array as you
create them), then in the finally iterate startedClients and call the client's
shutdown method (client.close() or client.dispose() as available) to ensure
Experimental_StdioMCPTransport stdio pipes are cleaned up; also ensure
addCodeBlockTool and addMarkdownBlockTool registration
(context.addAndExecuteCodeBlock / context.addMarkdownBlock) happens after
clients/tools are successfully created so partial startups are cleaned up
consistently.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/runtime-core/src/agent-handler.ts`:
- Around line 69-73: The mapping currently only overrides a block when
collectedOutputs.get(block.id) exists and has outputs.length > 0, which allows
stale block.outputs to persist when an entry exists with outputs: []; change the
condition so any presence of an entry in collectedOutputs for block.id (i.e.,
outputs != null / outputs !== undefined) causes the block to be returned with
outputs set to outputs.outputs (including the empty array); leave the original
block untouched only when there is no map entry for that id (refer to the
blocks.map callback, collectedOutputs.get(block.id), and the outputs.outputs
property).
- Around line 15-23: Update the callback result types so successful and failed
tool returns are serializable and include the execution output: replace
AddAndExecuteCodeBlockResult with a discriminated union where success: true
includes a required executionText (string) carrying stdout/stderr/results, and
success: false includes a SerializableError type (e.g., { message: string;
name?: string; stack?: string; code?: string }) instead of raw Error; do the
same for AddMarkdownBlockResult (e.g., success true includes renderedText or
content string). Then update the AgentBlockContext signatures for
addAndExecuteCodeBlock and addMarkdownBlock to return the new result types so
the agent always receives serializable output and meaningful error information.

---

Outside diff comments:
In `@packages/runtime-core/src/agent-handler.ts`:
- Around line 171-203: The MCP clients and their tool enumeration are created
outside the try/finally, so a failure partway leaves open stdio transports; move
the whole client startup and tools collection into the protected scope and track
started clients incrementally (e.g., push each created client from
mergedMcpConfig.map/createMCPClient into a local startedClients array as you
create them), then in the finally iterate startedClients and call the client's
shutdown method (client.close() or client.dispose() as available) to ensure
Experimental_StdioMCPTransport stdio pipes are cleaned up; also ensure
addCodeBlockTool and addMarkdownBlockTool registration
(context.addAndExecuteCodeBlock / context.addMarkdownBlock) happens after
clients/tools are successfully created so partial startups are cleaned up
consistently.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 84e86e00-c5ff-4e1a-8d96-c6677c2b1b9e

📥 Commits

Reviewing files that changed from the base of the PR and between f3c19fb and f70ded4.

📒 Files selected for processing (1)
  • packages/runtime-core/src/agent-handler.ts

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
packages/runtime-core/src/agent-handler.ts (2)

173-184: 🧹 Nitpick | 🔵 Trivial

MCP transport failures will reject the entire agent run.

If any MCP server fails to connect (bad command, missing binary), Promise.all rejects and no cleanup runs. Consider graceful degradation or wrapping each client creation.

♻️ Possible approach
-  const mcpClients = await Promise.all(
-    mergedMcpConfig.map(s =>
+  const mcpClientResults = await Promise.allSettled(
+    mergedMcpConfig.map(s =>
       createMCPClient({
         transport: new Experimental_StdioMCPTransport({
           command: s.command,
           args: s.args,
           env: resolveEnvVars(s.env),
           stderr: 'pipe',
         }),
       })
     )
   )
+  const mcpClients = mcpClientResults
+    .filter((r): r is PromiseFulfilledResult<Awaited<ReturnType<typeof createMCPClient>>> => r.status === 'fulfilled')
+    .map(r => r.value)
+  const failedCount = mcpClientResults.filter(r => r.status === 'rejected').length
+  if (failedCount > 0) {
+    context.onLog?.(`[agent] ${failedCount} MCP server(s) failed to connect`)
+  }
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/runtime-core/src/agent-handler.ts` around lines 173 - 184, The
current use of Promise.all over mergedMcpConfig to createMCPClient instances
means a single Experimental_StdioMCPTransport failure will reject the entire
agent run and skip cleanup; change the creation to handle per-client failures
(e.g., map mergedMcpConfig to individual try/catch or Promise.allSettled,
logging errors for failed createMCPClient calls and only collecting successful
clients into mcpClients) so that createMCPClient/Experimental_StdioMCPTransport
failures degrade gracefully and allow normal cleanup and operation with
remaining clients.

158-161: ⚠️ Potential issue | 🔴 Critical

Replace gpt-5 with an available model name.

The model 'gpt-5' does not exist. As of March 2025, the latest available OpenAI model is GPT-4.5. Update the fallback to use an actual model identifier.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/runtime-core/src/agent-handler.ts` around lines 158 - 161, The
fallback model string is invalid: update the model selection logic that sets the
modelName (using block.metadata.deepnote_agent_model and
process.env.OPENAI_MODEL) to use a real OpenAI model identifier instead of
"gpt-5" — replace that fallback with the current available model (e.g.,
"gpt-4.5") so modelName resolves to a valid model when deepnote_agent_model ===
'auto' and OPENAI_MODEL is unset.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@packages/runtime-core/src/agent-handler.ts`:
- Around line 173-184: The current use of Promise.all over mergedMcpConfig to
createMCPClient instances means a single Experimental_StdioMCPTransport failure
will reject the entire agent run and skip cleanup; change the creation to handle
per-client failures (e.g., map mergedMcpConfig to individual try/catch or
Promise.allSettled, logging errors for failed createMCPClient calls and only
collecting successful clients into mcpClients) so that
createMCPClient/Experimental_StdioMCPTransport failures degrade gracefully and
allow normal cleanup and operation with remaining clients.
- Around line 158-161: The fallback model string is invalid: update the model
selection logic that sets the modelName (using
block.metadata.deepnote_agent_model and process.env.OPENAI_MODEL) to use a real
OpenAI model identifier instead of "gpt-5" — replace that fallback with the
current available model (e.g., "gpt-4.5") so modelName resolves to a valid model
when deepnote_agent_model === 'auto' and OPENAI_MODEL is unset.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 8a0a03fc-f6af-401c-820d-23b7f0abaa6f

📥 Commits

Reviewing files that changed from the base of the PR and between f70ded4 and c85033d.

📒 Files selected for processing (1)
  • packages/runtime-core/src/agent-handler.ts

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/runtime-core/src/agent-handler.ts (1)

171-205: ⚠️ Potential issue | 🟠 Major

Clean up MCP clients if setup fails.

Cleanup only starts at Line 244. If one createMCPClient() or client.tools() call rejects here, any earlier stdio transports stay alive because they were created before the try/finally scope. Move MCP setup into the cleanup scope, or close the clients accumulated so far before rethrowing.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/runtime-core/src/agent-handler.ts` around lines 171 - 205, The MCP
client setup can leak processes if createMCPClient() or client.tools() rejects;
ensure partial clients are closed on failure by wrapping the creation and
tools-fetching in a try/catch/finally and, on any error, iterate the
already-created mcpClients and call their shutdown/close/dispose method (e.g.,
client.close() or client.shutdown(), using the actual API) to stop underlying
Experimental_StdioMCPTransport processes before rethrowing; similarly, ensure
any created clients are closed in the existing cleanup/finally block so
mergedMcpConfig, createMCPClient, mcpClients, mcpToolSets and client.tools()
failures do not leave stray stdio transports running.
♻️ Duplicate comments (1)
packages/runtime-core/src/execution-engine.ts (1)

252-286: ⚠️ Potential issue | 🟠 Major

Keep the full result for agent-inserted code blocks.

Line 283 collapses successful executions to { success: true }, the failure path still returns raw Error objects, and Lines 332-339 emit every inserted block as success: true anyway. Thrown executions also never reach onBlockDone because nothing is recorded in blockOutputs. Keep one serializable execution result and reuse it for both the tool response and host callbacks.

Also applies to: 323-339

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/runtime-core/src/execution-engine.ts` around lines 252 - 286, The
addAndExecuteCodeBlock function currently discards successful execution details
and only returns {success: true} while errors return raw Error objects and
failed runs aren't recorded in blockOutputs/collectedOutputs, so change
addAndExecuteCodeBlock to capture the full, serializable execution result from
kernel.execute (including outputs and executionCount and a serializable
error/message on failure), store that same result into blockOutputs and
collectedOutputs for every run (success or failure), and return that single
unified result structure so the tool response and any host callbacks (e.g.,
onBlockDone consumers) consume the exact same serializable execution result.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/runtime-core/src/execution-engine.ts`:
- Around line 236-242: Replace direct reads of process.env for agent provider
settings with the runtime-provided environment in RuntimeConfig.env: use the
existing runtimeConfig.env (the env passed into the engine at RuntimeConfig.env)
to obtain OPENAI_API_KEY and related overrides instead of process.env (affecting
the apiKey check that currently reads process.env.OPENAI_API_KEY and the
forwarded context created around lines 310-315). Ensure the error message
reflects the actual env source and mentions available overrides (e.g.,
OPENAI_API_KEY, OPENAI_BASE_URL, model override keys) when missing, and update
the logic that builds the forwarded context to copy relevant keys from
runtimeConfig.env so embedders can supply base URL and model overrides without
mutating process state.

---

Outside diff comments:
In `@packages/runtime-core/src/agent-handler.ts`:
- Around line 171-205: The MCP client setup can leak processes if
createMCPClient() or client.tools() rejects; ensure partial clients are closed
on failure by wrapping the creation and tools-fetching in a try/catch/finally
and, on any error, iterate the already-created mcpClients and call their
shutdown/close/dispose method (e.g., client.close() or client.shutdown(), using
the actual API) to stop underlying Experimental_StdioMCPTransport processes
before rethrowing; similarly, ensure any created clients are closed in the
existing cleanup/finally block so mergedMcpConfig, createMCPClient, mcpClients,
mcpToolSets and client.tools() failures do not leave stray stdio transports
running.

---

Duplicate comments:
In `@packages/runtime-core/src/execution-engine.ts`:
- Around line 252-286: The addAndExecuteCodeBlock function currently discards
successful execution details and only returns {success: true} while errors
return raw Error objects and failed runs aren't recorded in
blockOutputs/collectedOutputs, so change addAndExecuteCodeBlock to capture the
full, serializable execution result from kernel.execute (including outputs and
executionCount and a serializable error/message on failure), store that same
result into blockOutputs and collectedOutputs for every run (success or
failure), and return that single unified result structure so the tool response
and any host callbacks (e.g., onBlockDone consumers) consume the exact same
serializable execution result.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 41513acb-7b98-4c35-b5d0-ecd19248eb98

📥 Commits

Reviewing files that changed from the base of the PR and between c85033d and 17ad3e5.

📒 Files selected for processing (3)
  • packages/cli/src/commands/run.ts
  • packages/runtime-core/src/agent-handler.ts
  • packages/runtime-core/src/execution-engine.ts

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

♻️ Duplicate comments (2)
packages/runtime-core/src/execution-engine.ts (2)

261-295: ⚠️ Potential issue | 🟠 Major

Keep the full inserted-block result and emit it inline.

Lines 261-295 drop success/error and buffer callbacks until after executeAgentBlock() returns. A result.success === false is later reported as success: true at Line 344, and any inserted block run before a later agent exception never reaches onOutput or onBlockDone.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/runtime-core/src/execution-engine.ts` around lines 261 - 295, The
addAndExecuteCodeBlock helper currently buffers execution results and defers
emitting them, causing failures (result.success === false) to be misreported
later; modify addAndExecuteCodeBlock to emit the full inserted-block result
immediately after kernel.execute returns or throws: after computing
result.outputs and result.executionCount, call the existing onOutput/new-output
handler (the same callback used elsewhere) with the outputs and then call
onBlockDone/onBlockFinished with either success: true or success: false and the
Error (do this in both the successful path and the catch path), and still update
collectedOutputs and blockOutputs as now; ensure you do not rely on later
post-processing in executeAgentBlock to flip success to true—preserve and pass
the original success/error through immediately so inserted block runs reach
onOutput and onBlockDone inline.

245-250: ⚠️ Potential issue | 🟠 Major

Source agent auth from RuntimeConfig.env.

Line 245 still reads process.env.OPENAI_API_KEY directly. That bypasses the runtime-configured env, so embedders have to mutate global process state, and the fallback text is misleading because OPENAI_BASE_URL still hits this branch.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@packages/runtime-core/src/execution-engine.ts` around lines 245 - 250,
Replace the direct process.env lookup for OPENAI_API_KEY with the
runtime-configured env (use RuntimeConfig.env.OPENAI_API_KEY) in the block that
currently throws the Error, and update the thrown error text to reflect sourcing
from RuntimeConfig.env (remove or correct the misleading OPENAI_BASE_URL
fallback mention); i.e., change the const apiKey assignment to read from
RuntimeConfig.env and adjust the throw new Error message in the same block to
instruct configuring OPENAI_API_KEY via RuntimeConfig.env or equivalent embedder
configuration.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@packages/runtime-core/src/execution-engine.test.ts`:
- Around line 836-847: Add new Vitest cases in execution-engine.test.ts that
exercise the ExecutionEngine context helpers: call
context.addAndExecuteCodeBlock(...) and context.addMarkdownBlock(...) (or the
instance method names used in the suite) and assert they insert blocks into the
engine's store, invoke kernel.execute (mock the kernel.execute and assert it was
called with the expected code, language and context), emit the same
callbacks/events the existing tests expect (spy/mock any callback handlers used
in the suite), and verify error propagation by having the mocked kernel.execute
reject to ensure the helper surfaces the failure; reuse existing mocks like
mockExecuteAgentBlock where appropriate and add new mocks/spies for
kernel.execute and block insertion assertions.

---

Duplicate comments:
In `@packages/runtime-core/src/execution-engine.ts`:
- Around line 261-295: The addAndExecuteCodeBlock helper currently buffers
execution results and defers emitting them, causing failures (result.success ===
false) to be misreported later; modify addAndExecuteCodeBlock to emit the full
inserted-block result immediately after kernel.execute returns or throws: after
computing result.outputs and result.executionCount, call the existing
onOutput/new-output handler (the same callback used elsewhere) with the outputs
and then call onBlockDone/onBlockFinished with either success: true or success:
false and the Error (do this in both the successful path and the catch path),
and still update collectedOutputs and blockOutputs as now; ensure you do not
rely on later post-processing in executeAgentBlock to flip success to
true—preserve and pass the original success/error through immediately so
inserted block runs reach onOutput and onBlockDone inline.
- Around line 245-250: Replace the direct process.env lookup for OPENAI_API_KEY
with the runtime-configured env (use RuntimeConfig.env.OPENAI_API_KEY) in the
block that currently throws the Error, and update the thrown error text to
reflect sourcing from RuntimeConfig.env (remove or correct the misleading
OPENAI_BASE_URL fallback mention); i.e., change the const apiKey assignment to
read from RuntimeConfig.env and adjust the throw new Error message in the same
block to instruct configuring OPENAI_API_KEY via RuntimeConfig.env or equivalent
embedder configuration.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 1c227c24-a059-482c-bb5a-2a2b36e19e9e

📥 Commits

Reviewing files that changed from the base of the PR and between 17ad3e5 and b6bc0e7.

📒 Files selected for processing (2)
  • packages/runtime-core/src/execution-engine.test.ts
  • packages/runtime-core/src/execution-engine.ts

Base automatically changed from agent-poc-2 to main March 19, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants