Skip to content

feat: AI prompt management dashboard and enhanced span inspectors#3244

Open
ericallam wants to merge 24 commits intomainfrom
feature/tri-7888-ai-prompts
Open

feat: AI prompt management dashboard and enhanced span inspectors#3244
ericallam wants to merge 24 commits intomainfrom
feature/tri-7888-ai-prompts

Conversation

@ericallam
Copy link
Member

@ericallam ericallam commented Mar 21, 2026

  • Full prompt management UI: list, detail, override, and version management for AI prompts defined with prompts.define()
  • Rich AI span inspectors for all AI SDK operations with token usage, messages, and prompt context
  • Real-time generation tracking with live polling and filtering

Prompt management

Define prompts in your code with prompts.define(), then manage versions and overrides from the dashboard without redeploying:

import { task, prompts } from "@trigger.dev/sdk";
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";

const supportPrompt = prompts.define({
  id: "customer-support",
  model: "gpt-4o",
  variables: z.object({
    customerName: z.string(),
    plan: z.string(),
    issue: z.string(),
  }),
  content: `You are a support agent for Acme SaaS.
Customer: {{customerName}} ({{plan}} plan)
Issue: {{issue}}
Respond with empathy and precision.`,
});

export const supportTask = task({
  id: "handle-support",
  run: async (payload) => {
    const resolved = await supportPrompt.resolve({
      customerName: payload.name,
      plan: payload.plan,
      issue: payload.issue,
    });

    const result = await generateText({
      model: openai(resolved.model ?? "gpt-4o"),
      system: resolved.text,
      prompt: payload.issue,
      ...resolved.toAISDKTelemetry(),
    });

    return { response: result.text };
  },
});

The prompts list page shows each prompt with its current version, model, override status, and a usage sparkline over the last 24 hours.

From the prompt detail page you can:

  • Create overrides to change the prompt template or model without redeploying. Overrides take priority over the deployed version when prompt.resolve() is called.
  • Promote any code-deployed version to be the current version
  • Browse generations across all versions with infinite scroll and live polling for new results
  • Filter by version, model, operation type, and provider
  • View metrics (total generations, avg tokens, avg cost, latency) broken down by version

AI span inspectors

Every AI SDK operation now gets a custom inspector in the run trace view:

  • ai.generateText / ai.streamText — Shows model, token usage, cost, the full message thread (system prompt, user message, assistant response), and linked prompt details
  • ai.generateObject / ai.streamObject — Same as above plus the JSON schema and structured output
  • ai.toolCall — Shows tool name, call ID, and input arguments
  • ai.embed — Shows model and the text being embedded

For generation spans linked to a prompt, a "Prompt" tab shows the prompt metadata, the input variables passed to resolve(), and the template content from the prompt version.

All AI span inspectors include a compact timestamp and duration header.

Other improvements

  • Resizable panel sizes now persist across page refreshes (patched @window-splitter/state to fix snapshot restoration)
  • Run page panels also persist their sizes
  • Fixed <div> inside <p> DOM nesting warnings in span titles and chat messages
  • Added Operations and Providers filters to the AI metrics dashboard

Screenshots

CleanShot 2026-03-21 at 10 14 17@2x CleanShot 2026-03-21 at 10 15 37@2x CleanShot 2026-03-21 at 10 15 54@2x

@changeset-bot
Copy link

changeset-bot bot commented Mar 21, 2026

🦋 Changeset detected

Latest commit: 8cd61cf

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 29 packages
Name Type
@trigger.dev/sdk Patch
@trigger.dev/python Patch
@internal/sdk-compat-tests Patch
d3-chat Patch
references-d3-openai-agents Patch
references-nextjs-realtime Patch
references-realtime-hooks-test Patch
references-realtime-streams Patch
references-telemetry Patch
@trigger.dev/build Patch
@trigger.dev/core Patch
@trigger.dev/react-hooks Patch
@trigger.dev/redis-worker Patch
@trigger.dev/rsc Patch
@trigger.dev/schema-to-json Patch
@trigger.dev/database Patch
@trigger.dev/otlp-importer Patch
trigger.dev Patch
@internal/cache Patch
@internal/clickhouse Patch
@internal/llm-pricing Patch
@internal/redis Patch
@internal/replication Patch
@internal/run-engine Patch
@internal/schedule-engine Patch
@internal/testcontainers Patch
@internal/tracing Patch
@internal/tsql Patch
@internal/zod-worker Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Mar 21, 2026

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

Implements end-to-end prompt management: adds Prompt and PromptVersion Prisma models and migrations; extends ClickHouse LLM metrics schema and ingestion with prompt fields; introduces PromptService, PromptPresenter, API routes (list, resolve, promote, override lifecycle, generations), and ClickHouse queries; adds frontend pages/components (prompts list, prompt detail, filters, dashboard integration, span inspectors, editors); enriches events/metrics with prompt telemetry; integrates prompts into worker indexing, SDK (define/resolve prompts), CLI/MCP tools, resource catalog, and related schemas/types across the codebase.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120+ minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 20.79% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely summarizes the primary changes: AI prompt management dashboard and enhanced span inspectors for AI operations.
Description check ✅ Passed The PR description is comprehensive with detailed feature explanations, code examples, usage instructions, and three screenshots, but the required checklist items are not checked off and no testing section details are provided.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feature/tri-7888-ai-prompts

coderabbitai[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

@ericallam ericallam marked this pull request as ready for review March 22, 2026 07:28
devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

…reactivateOverride, use nullish coalescing in updateOverride
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

Copy link
Contributor

@devin-ai-integration devin-ai-integration bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 new potential issues.

View 18 additional findings in Devin Review.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants