Some projects teach you more than others. This was one of them.

Over the past year, our team built an AI agent capable of querying a complex, on-premises SQL environment for a large enterprise client. What started as a focused, scoped project gradually grew in complexity and ambition — and by the end, we were deep in the architecture, rebuilding core components and shipping the system to UAT. Here's what we learned.

## The Environment We Were Working With

Before talking about the agent, it's worth understanding the data environment — because that context shapes every decision we made.

The database was a legacy on-premises SQL Server system composed of multiple linked views spanning several business domains. Behind-the-scenes dependencies made it fragile: joins failed unexpectedly, performance degraded on complex queries, and there was no fuzzy search or text indexing to handle real-world inputs like misspelled names or partial matches.

On top of that, every query the agent generated had to respect role-based access rules across a strict organizational hierarchy, each level with a different data scope. Permission filtering on dynamic, AI-generated SQL is not a solved problem. Getting it wrong in an enterprise environment has real consequences.

This wasn't a clean sandbox. It was the kind of environment most production AI systems actually live in.

## What the Foundation Got Right

Several early decisions held up well over time and didn't need revisiting:

- **Permission enforcement in code, not prompts.** Role-based access was never left to the model to enforce. A separate authorization layer resolved each user's data scope before any query ran — pulling dynamically from the database, with a config file as fallback. This was the right call from day one.
- **Per-data-source tools.** Each business domain had its own plugin responsible for generating and executing queries. For isolated use cases, this worked cleanly and kept concerns separated.
- **Partial data caching.** Only a fixed page of rows was served to the agent at a time, with the rest stored server-side. This kept latency low, reduced token usage, and made the system feel fast to users.
- **User-specific export caches.** A lightweight system let users retrieve historical query results as Excel files or trigger fresh queries on demand — a small UX detail that made a meaningful difference in daily use.
- **Comprehensive logging.** Every tool call, query, and error was tracked using OpenTelemetry from the start — creating a foundation for debugging and building trust with stakeholders.

## Where Things Got Complicated

As the system matured and user expectations grew, a few structural limitations became clear.

The prompt design was overly rigid — the agent struggled with queries that crossed multiple data sources or didn't match the patterns it had been designed to expect. Five separate domain-specific plugins meant the agent had to know in advance which domain a question belonged to, which broke down for anything cross-domain. The permission filtering, while present, was implemented as basic `WHERE` clause injection — fragile the moment a query has its own CTEs, subqueries, or aggregates.

These weren't failures of effort. They were the natural result of building in an environment with limited visibility and a lot of unknowns. But they created a ceiling the system kept hitting.

## The Shift: A Unified Explorer Agent

The most significant improvement came from consolidating the five domain plugins into one unified agent — built on Semantic Kernel with Azure OpenAI — using what we call the explorer-first pattern.

Instead of routing questions to pre-built domain generators, the agent now discovers what it needs before writing SQL. It exposes a small, focused toolset to the orchestrator:

- **List Tables** — returns a catalogue of all available data sources with descriptions and join keys. Also injects role-scoped context for the current user so the agent understands what data is in scope.
- **Get Schema** — returns column definitions and sample rows for a specific source.
- **Get Distinct Values** — returns unique values for a column with optional search, solving the fuzzy-match problem for name lookups.
- **Resolve Name to Code** — translates a human-readable entity name into its internal identifier code, preventing the agent from ever filtering on name columns directly.
- **Execute SQL** — validates, secures, and runs the query with full retry logic.

The agent calls these in sequence: discover → inspect schema → resolve filter values → generate SQL → validate and secure → execute. Upgrading to GPT-5.2 made this pattern significantly more reliable — stronger instruction-following and contextual reasoning meant fewer hallucinated column names and better multi-source join handling.

## Smarter Permission Control with Security CTEs

One of the most important architectural changes was how we rethought permission enforcement at the query level.

The old approach injected `WHERE` clauses directly into model-generated SQL. This is fragile — it breaks the moment the query has its own CTEs, uses self-joins, or has a complex aggregation structure.

The new approach uses what we call the **Security CTE Prefix** pattern. Instead of modifying the model's SQL, we prepend pre-filtered security CTEs that replace raw data source references entirely. The model's query is treated as a black box:

```sql
-- Model generates:
SELECT department, COUNT(*) AS headcount
FROM employees
GROUP BY department

-- System transforms to:
WITH _sec_employees AS (
  SELECT * FROM employees
  WHERE department IN ('dept_a', 'dept_b')
)
SELECT department, COUNT(*) AS headcount
FROM _sec_employees
GROUP BY department
```

Every data source reference in the model's SQL — however it's written — gets replaced with the pre-filtered CTE. CTEs, subqueries, self-joins, and aggregates all work safely without special cases. There's also a post-execution safety net: after results return, any row outside the user's authorized scope is stripped before the data reaches the model.

## A Second Review Layer for Complex Queries

For queries that cross multiple data sources, use CTEs, or contain nested selects, we added an optional LLM-based SQL review step. Before execution, the generated SQL is passed to a second model call with a structured prompt asking it to return a simple decision: approve, revise, or block.

This reviewer doesn't need to catch everything — the deterministic validator and security CTE filter are the real safety net. The reviewer's job is correctness: catching duplicate inflation from missing `DISTINCT`, wrong source usage, or broken aggregations. It runs with low reasoning effort and a tight timeout so it doesn't slow the user experience.

## Observability

A reliable agent isn't just one that produces good outputs — it's one you can inspect and verify.

Every database call is wrapped in an OpenTelemetry span with the query, row count, and error state attached. Semantic Kernel's own traces flow into the same pipeline. This gave us end-to-end visibility from the user's question through to the final SQL execution — which proved critical during high-stakes UAT sessions with external stakeholders.

## What I'd Do Differently from Day One

If I were starting fresh:

- **Start with data modernization.** Unified schema design, quality constraints, and proper join performance via a data lake rather than fighting a legacy view structure throughout development.
- **Define agent-friendly metadata upfront.** Column descriptions, join keys, and domain rules should live in source control from the start — not discovered iteratively through production failures.
- **Use the explorer pattern from day one.** Domain-specific plugins feel clean early and become a maintenance burden fast. A single agent that discovers schema dynamically scales much better.
- **Enforce access control natively at the infrastructure level.** Application-layer CTE injection is a solid fallback for environments you don't fully control, but native row-level security is a stronger long-term foundation.
- **Add query-result caching early.** Repeated queries on stable data are an easy performance win that meaningfully improves the user experience.

## A Note on Where AI Agents Are Heading

There's a lot of excitement right now about fully autonomous agents — systems that can plan, act, and iterate with minimal human intervention. That's a real and important direction.

But in regulated industries and enterprise environments, the properties that matter most are different: auditability, permission enforcement, predictable behavior, and the ability to explain exactly how an answer was reached. The patterns here — structured tooling, CTE-based access control, deterministic validation, full execution tracing — aren't legacy thinking. They're what production AI requires when the stakes are real and the data is sensitive.

The agent frameworks will keep evolving. The fundamentals won't.

## The Open-Source Version

The core patterns from this project — the explorer-first toolset, the Security CTE Prefix, and the read-only validator — are available as an open-source reference implementation at [github.com/tomvsaji/text-to-sql-agent](https://github.com/tomvsaji/text-to-sql-agent). That version uses LangGraph and a SQLite demo database rather than Semantic Kernel and SQL Server, but the security architecture and agent design transfer directly.

Lessons from building and hardening an enterprise text-to-SQL agent in a fragile on-prem SQL Server environment.

Building Reliable SQL Agents in Complex, Legacy Environments

Composer 2 did not just show up as another coding model launch. It showed that a product team can get surprisingly close to Claude Opus and Codex-class coding quality without paying frontier-model prices on every request. For AI engineers, that makes Composer 2 interesting for a reason beyond benchmarks: it is a strong case study in specialization, reinforcement learning in real environments, and infrastructure designed around one narrow but valuable workflow.

## 1. Cost and performance: a new default for coding

Cursor is pricing Composer 2 like a default tool, not a premium upsell. It split the model into two SKUs with the same stated intelligence level, one optimized for lower cost and one for lower latency.

- **Composer 2 (Standard):** $0.50 input and $2.50 output per 1M tokens, positioned as frontier-level coding at lower cost.
- **Composer 2 (Fast):** $1.50 input and $7.50 output per 1M tokens, with the same intelligence but lower latency and default placement inside Cursor.
- **Claude Sonnet 4.6:** about $3 input and $15 output per 1M tokens, with stronger general breadth but materially higher cost.
- **Claude Opus 4.6:** about $5 input and $25 output per 1M tokens as a premium higher-cost tier.
- **GPT-5.4-class models:** typically closer to Opus-tier pricing, with higher scores on some benchmarks but at meaningfully higher cost.

On coding-focused benchmarks, Composer 2 lands in the same broad band as Claude and GPT-style coding agents while costing much less:

- Composer 2: 61.3 on CursorBench, 61.7 on Terminal-Bench 2.0, and 73.7 on SWE-bench Multilingual.
- Claude Opus 4.6: around 58 on Terminal-Bench 2.0 in the comparisons cited around the launch.
- GPT-5.4-class systems: higher on some agent benchmarks, but at meaningfully higher price points.

The practical takeaway for teams is simple: Composer 2 is aiming to deliver near-frontier coding quality at pricing closer to an everyday developer tool than a premium reasoning model.

## 2. What Cursor did differently in training

Cursor did not try to beat OpenAI or Anthropic at general intelligence. It optimized for one narrower target: agentic coding inside real repositories with tools, files, terminals, and long-running tasks.

### 2.1. Start from a strong base, then specialize aggressively

Composer 2 is built on top of Moonshot’s Kimi K2.5, a strong multilingual and code-capable base model. Cursor then appears to have pushed hard on continued pretraining and coding-specific adaptation.

That specialization matters. Instead of spending most of the budget on a generic web-scale assistant, Cursor concentrated on code, repositories, terminals, and agent transcripts. The result is a model distribution that looks much more like real software work than broad internet text.

### 2.2. RL in a real IDE instead of a toy benchmark

The biggest architectural difference is that Cursor trains and evaluates in environments that look much closer to the actual product:

- RL rollouts run in Cursor-like environments with real repositories, file operations, semantic search, terminal access, tests, and background agents.
- The company says it scaled to hundreds of thousands of sandboxed coding environments, reusing infrastructure that already powers production agent workflows.
- Rewards are not just about whether tests pass. They also appear to incorporate speed, tool efficiency, concise interaction patterns, and the ability to maintain abstractions over longer horizons.

That is a very different optimization loop from training a general chat model and only later wrapping it in tools. Cursor is effectively training much closer to the environment where the model is supposed to work.

### 2.3. Long-horizon work through self-summarization

Cursor also introduced self-summarization so the model can compress long coding sessions into short persistent summaries that preserve the important state across many steps.

That matters for repository-scale work. In real engineering tasks, the challenge is often not just producing the next answer; it is staying coherent across dozens or hundreds of actions, file changes, tool calls, and partial plans. By training on summarization and long-horizon workflows, Cursor appears to be optimizing for exactly that failure mode.

### 2.4. Infrastructure choices that reinforce the product

Composer 2’s infrastructure choices are also tightly aligned with the product:

- MoE architecture and MXFP8 precision help reduce the amount of dense compute needed for training and inference.
- PyTorch and Ray-style RL orchestration support large numbers of concurrent rollouts across sandboxed environments.
- KV-cache-aware serving and cache-friendly interaction patterns make long iterative sessions cheaper to run.

This is an important pattern for AI engineers: infrastructure is not separate from product strategy. In Composer 2’s case, the serving stack, RL system, and developer experience all reinforce the same core loop of coding, testing, and iterating quickly.

## 3. Why Composer 2 can rival Claude and Codex at lower cost

Composer 2’s advantage does not look like mysterious model magic. It looks like stack alignment.

- **Narrow domain focus:** the model is optimized for programming tasks rather than for every kind of general-purpose reasoning.
- **Shared runtime across training and product:** the same kinds of environments used for RL also power the product experience, so improvements transfer more directly.
- **Evaluation tied to actual coding work:** Cursor benchmarks against coding-heavy suites like CursorBench, Terminal-Bench 2.0, and SWE-bench Multilingual, then prices aggressively against more general frontier models.

Claude and Codex-style systems still have advantages in broader cross-domain reasoning. But they also carry the cost of being broad systems. Composer 2 benefits from not paying for capabilities it does not need to emphasize, then reinvesting that budget into coding-specific training, RL, and infrastructure.

## 4. Lessons for AI engineers

Composer 2 is not just a product launch. It is a useful blueprint for people building specialized AI systems.

### 4.1. Optimize for workflows, not generic intelligence

Pick a vertical where outcomes matter and where you can define the loop clearly. Then align data, rewards, tools, evaluation, and product design around that loop instead of trying to maximize vague general-purpose intelligence.

### 4.2. Train in the environment you actually ship

If your product depends on tools, logs, tests, files, APIs, or stateful workflows, your training and evaluation environment should expose those same elements. Toy environments can be useful early on, but production-like environments create much more relevant learning signals.

A practical first step is to instrument your existing system:

- Log tasks, tool calls, diffs, and test results.
- Turn successful trajectories into supervised data.
- Use partial success and failure recovery as reward signals for preference optimization or RL.

### 4.3. Reward speed and clean execution, not just correctness

Passing tests is not enough. Good coding agents also need to avoid wasted tool calls, preserve abstractions, recover from errors, and reach a good solution quickly. Those qualities can and should be represented in the reward design.

### 4.4. Treat infrastructure as strategic leverage

Serving, training, and application architecture should be designed together. Cursor’s VM scheduler, sandboxing, rollout infrastructure, and cache strategy all appear to support both model improvement and product economics at the same time.

### 4.5. Build your own domain benchmark

One of the most transferable lessons is to create an internal benchmark grounded in real tasks from your own environment. For a coding team, that could mean multi-file bug fixes, refactors, migrations, and test repair. For another domain, it should look like that domain’s actual work, not generic public benchmarks.

That creates a closed loop: model changes, benchmark results, shipping decisions, and production feedback all inform one another. That loop is often more valuable than squeezing out a few extra points on a generic leaderboard.

In that sense, Composer 2 is a reminder that the most efficient AI systems are often not the most general ones. They are the ones built with unusually tight alignment between model, environment, rewards, infrastructure, and product goals.

:::details References
- [Composer 2: Opus 4.6 and GPT-5.4 Just Got Beaten by a ...](https://apidog.com/blog/cursor-composer-2/)
- [Cursor acknowledges its new low-cost coding model has Chinese ...](https://www.businessinsider.com/cursor-composer-chinese-model-kimi-moonshot-ai-coding-low-cost-2026-3)
- [Cursor's new Composer 2 just beat Claude Opus at coding — and it's 10x cheaper](https://www.reddit.com/r/GoogleGemini/comments/1ry86e9/cursors_new_composer_2_just_beat_claude_opus_at/)
- [Cursor's Self-Developed Model Outperforms Opus 4.6 with Drastic ...](https://eu.36kr.com/en/p/3730749500539142)
- [Compare Claude Sonnet 4 vs. Composer 2 in 2026](https://slashdot.org/software/comparison/Claude-Sonnet-4-vs-Composer-2/)
- [Holy: Composer 2 is now live in Cursor, pairing frontier ...](https://x.com/kimmonismus/status/2034667869816979645)
- [Introducing Composer 2](https://cursor.com/blog/composer-2)
- [Composer: Building a fast frontier model with RL](https://cursor.com/blog/composer)
- [Cursor Composer 2: Features, Pricing, Benchmarks, and ...](https://explore.n1n.ai/blog/cursor-composer-2-features-pricing-benchmarks-2026-03-20)
- [Introducing Composer 2 - Cursor Community Forum](https://forum.cursor.com/t/introducing-composer-2/155288)
- [Cursor Launches Composer 2: Outperforms Opus 4.6 in ...](https://www.kucoin.com/news/flash/cursor-launches-composer-2-outperforms-opus-4-6-in-programming-benchmarks-pricing-drops-to-14-of-previous-model)
- [Cursor Composer 2 takes on Anthropic and OpenAI with a ...](https://topaiproduct.com/2026/03/19/cursor-composer-2-takes-on-anthropic-and-openai-with-a-0-50-m-token-coding-model-and-the-benchmarks-back-it-up/)
- [Cursor Composer 2 Review 2026: Benchmarks & Pricing - StackBuilt](https://stackbuiltai.com/cursor-composer-2-review-2026/)
- [Cursor's Composer 2 model just hit -- Time to change from Opus to Composer 2?](https://www.reddit.com/r/openclaw/comments/1rz7jli/cursors_composer_2_model_just_hit_time_to_change/)
- [Cursor's new coding model Composer 2 is here](https://venturebeat.com/technology/cursors-new-coding-model-composer-2-is-here-it-beats-claude-opus-4-6-but)
- [How Cursor Built Composer 2 on Top of Kimi K2.5](https://getaibook.com/blog/cursor-composer-2-is-built-on-kimi-k2-5)
- [The application layer strikes back: Cursor's custom LLM](https://bdtechtalks.substack.com/p/the-application-layer-strikes-back)
- [Building Cursor Composer: A Fast, Intelligent Agent-Based ...](https://www.zenml.io/llmops-database/building-cursor-composer-a-fast-intelligent-agent-based-coding-model-with-reinforcement-learning)
- [Sasha Rush on Building Cursor Composer and the Future ...](https://www.youtube.com/watch?v=vH5cGOxoXGY)
- [Claude SWE-Bench Performance](https://www.anthropic.com/engineering/swe-bench-sonnet)
- [Claude 3.5 Sonnet Complete Guide: AI Capabilities & Limits](https://galileo.ai/blog/claude-3-5-sonnet-complete-guide-ai-capabilities-analysis)
- [Introducing GPT-4.1 in the API](https://openai.com/index/gpt-4-1/)
- [Training Composer for longer horizons](https://cursor.com/blog/self-summarization)
- [Composer: Building a fast frontier model with RL (Simon Willison)](https://simonwillison.net/2025/Oct/29/cursor-composer/)
- [Master KV cache aware routing with llm-d for efficient AI inference](https://developers.redhat.com/articles/2025/10/07/master-kv-cache-aware-routing-llm-d-efficient-ai-inference)
- [KV cache offloading | LLM Inference Handbook](https://bentoml.com/llm/inference-optimization/kv-cache-offloading)
- [Boosting LLM Performance with Tiered KV Cache on Google ...](https://cloud.google.com/blog/topics/developers-practitioners/boosting-llm-performance-with-tiered-kv-cache-on-google-kubernetes-engine)
- [How Much Does Cursor Composer Cost?](https://www.cometapi.com/how-much-does-cursor-composer-cost/)
- [Stop Overpaying: Why Cursor Composer 2 Just Killed Claude Opus](https://www.youtube.com/watch?v=i6mqugBCp-E)
- [GPT-5 Codex vs Claude Sonnet 4.5 vs Grok Code vs Composer (Speed Test)](https://www.youtube.com/watch?v=Xvr-Osm_PDU)
:::

Why Cursor’s Composer 2 gets close to Claude and Codex-class coding quality at much lower cost, and what AI engineers can learn from its specialization, RL setup, and infra choices.

Composer 2 vs Claude & Codex: Why Cursor’s Model Is So Efficient (and What AI Engineers Can Learn)

Over the last year, AI assistants have gone from fun demos to tools we actually rely on. But most of them still live in a single chat box. Dreamer takes a different approach: it is building an operating system for **personal agents** that can live with you across your entire day, not just answer questions in a tab.

## What Is Dreamer?

Dreamer is a cloud platform where your main interface is a personal agent called **Sidekick**, surrounded by a growing ecosystem of other agents and tools.
You talk to Sidekick in natural language, and it in turn uses other agents such as calendar agents, research agents, coding agents, and reading agents to get work done on your behalf.
Under the hood, these agents are real software running in secure, isolated VMs, with access to tools, data sources, and each other through a common OS layer.

The Dreamer team describes it as “an operating system for AI agents and agentic apps,” taking lessons from Android, ChromeOS, and other platforms they previously worked on.
That OS metaphor is not just branding: it has primitives for agents, tools, permissions, logging, a database, and serverless functions, so you are effectively installing and composing intelligent apps, not just prompts.

## How Dreamer Helps With Everyday Life

Dreamer’s pitch is simple: instead of juggling dozens of apps, filters, and automations, you describe outcomes to Sidekick, and it orchestrates agents and tools to make them happen.

Here are a few ways that can play out in day-to-day life.

### 1. Your personal operations chief

You might say: `Get me ready for tomorrow.`
Sidekick can pull from calendar agents, email agents, and reading agents to build a single briefing with your meetings, relevant docs, and articles you saved that relate to the topics on your schedule.
Instead of bouncing between inbox, calendar, and notes, you get one coherent snapshot shaped around what you actually need to know.

Another example is: `Clean up my inbox and surface what matters.`
A mail agent can classify emails, summarize long threads, and highlight messages that need action, while a calendar agent suggests follow-up slots and drafts responses where appropriate.

### 2. Turning chores into background processes

Because agents can call each other, you can chain workflows without writing scripts.

A simple scenario the team highlights:

- You say: `Plan dinner for tonight and add the ingredients to my grocery list.`
- Sidekick uses a Recipe agent to pick something aligned with your preferences.
- It extracts ingredients and passes them to a Grocery List agent, which formats and stores them wherever you track shopping.

What you experience is a conversational request and a ready-to-shop list. What is actually happening is multiple agents talking through the OS.

### 3. A smarter reading and research flow

Imagine saying: `Whenever I star an article, summarize it and add key points to my second brain.`
In Dreamer, Sidekick can wire up a Personal News agent that discovers or ingests articles, along with a Read Later or Notes agent that stores content and structured summaries.

Because agents expose functions, the News agent can send an article directly into the Read Later agent without you copy-pasting anything.
You are effectively building custom pipelines for information digestion using natural language, not automation GUIs.

### 4. Help for non-technical and technical users

If you are not technical, you can stay in Sidekick and the agent gallery: discover agents, install them, and describe workflows you want, such as `When X happens, do Y and Z for me`.
If you are technical, Dreamer offers an SDK, CLI, logging, a database, and serverless functions so you can ship full-blown agentic apps that run arbitrary code in their VMs.

Crucially, the SDK is designed so that Sidekick and other coding agents can manipulate it too, meaning agents can help build and improve other agents.
That is where the `agent that builds agents` idea comes from, and it is a big part of why Dreamer feels different from typical agent builders.

## What Makes Dreamer Special?

Lots of products call themselves an agent OS or AI workspace. Dreamer stands out because it is opinionated about being an OS for intelligent software, not just a wrapper around an LLM.

### 1. Sidekick as the system agent

At the center of Dreamer is Sidekick, a system-level agent that knows about all your other agents, tools, and data, and mediates how they work together.
You can name it, shape its personality, and treat it as your long-term collaborator that understands your preferences and workflows.

Sidekick is also important for safety and control: other agents do not just run wild; they go through Sidekick and the OS permission model when they want to use tools or access data.

### 2. Composable agents with real capabilities

Every agent in Dreamer can expose **functions** to the rest of the system, which means any capability like `summarize this PDF`, `fetch from this API`, or `update this doc` can be reused by other agents or by Sidekick.
This turns agents into building blocks you can wire together, instead of isolated chatbots that force you to glue everything with manual steps.

The Dreamer team gives examples where content discovered by one agent can be passed directly into another agent built by someone else.
The OS provides the conventions and runtime that make this interoperability reliable.

### 3. A clear tools and permissions model

Dreamer treats tools as first-class: each tool is an MCP server plus optional skills, branding, and metadata that show up in the gallery.
Tools sit inside the permissions model, so the OS knows exactly which tools an agent might use, and you can inspect and approve those when you install someone else’s agent.

This is a meaningful shift from opaque plugin systems in some assistants. You do not just click `enable everything`; you see what is in play, who built it, and what each agent can do with it.

### 4. Full-stack, software 3.0 foundation

Rather than limiting you to no-code blocks, Dreamer lets builders deploy agents that:

- Run arbitrary deterministic code where that is best
- Call prompts into state-of-the-art LLMs
- Invoke sub-agents and Sidekick tasks that can manipulate the whole system

Agents run in secure, isolated VMs with physical data separation per user, giving a more robust foundation for serious personal software.
The system is designed so that everything you build is intelligence-native by default, which the team frames as `software 3.0`.

### 5. Community and ecosystem, not just a product

Dreamer is intentionally cultivating a community of builders: an early alpha group, builders-in-residence, and even cash prizes for useful agents and tools.
The goal is to make Dreamer feel like an app store where every app can talk to every other app, and where both hobbyists and professionals can publish and monetize their creations.

That ecosystem focus is key to the OS ambition. No single company can anticipate every niche workflow; you need thousands of people building agents that solve their own problems and then sharing them.

## How You Actually Use Dreamer Today

Dreamer is currently in beta with a waitlist, open to both technical and non-technical early adopters.
You access it primarily via a web app that works on modern browsers on Windows, macOS, and Linux, with Sidekick and agents available across devices and surfaces.

Once you are in, a typical journey looks like:

- Naming and personalizing your Sidekick
- Installing a few starter agents from the gallery, such as email, reading, coding, and planning agents
- Asking Sidekick to tie them into workflows that match your real life, such as `prepare me for tomorrow`, `optimize my reading`, or `help me with recruiting`

If you are a builder, you can drop down into the SDK and CLI to create custom agents, share them with the community, and eventually monetize them as the platform matures.

## Why Dreamer Matters

The Dreamer team is betting that 2026 is the year of the **personal agent**, and that the right abstraction is not `chat with a model`, but `live with a Sidekick that coordinates many agents on an OS built for them`.
If they are right, the way we use computers shifts: you spend less time clicking and configuring, and more time stating goals while a network of agents quietly handles the glue work.

For people overwhelmed by tools, tabs, and to-dos, that is a compelling vision: not just smarter apps, but a smarter environment that is actually shaped around you.

:::details References
- [David Singleton on Latent Space](https://www.latent.space/p/dreamer)
- [Dreamer launch post](https://blog.singleton.io/posts/2026-02-17-introducing-dreamer/)
- [Sidekick tour](https://www.youtube.com/watch?v=k07CJ8afVgQ)
- [StartupHub overview](https://www.startuphub.ai/ai-news/artificial-intelligence/2026/dreamer-building-ai-agents-for-everyone)
- [Dreamer community strategy](https://blog.singleton.io/posts/2026-03-17-community-is-a-product-decision/)
:::

A practical look at Dreamer, Sidekick, and why the platform is positioning itself as an operating system for personal AI agents.

Dreamer: The Personal Agent OS That Wants To Run Your Day

AI agents have moved from research demos to production systems that browse the web, call tools, operate UIs, and solve complex workflows end to end. As soon as you put an agent in front of real users or real data, one question becomes critical: **how do you evaluate it?**

In 2026, agent evaluation has become its own discipline. It combines classic ML metrics, new agentic benchmarks, and specialized tooling for tracing multi-step behavior. This post walks through a practical way to think about agent evaluation: what to measure, how to measure it, and which tools you can use today.

---

## What to measure

### 1. Task success and correctness

At the core, you want to know if the agent actually solves the tasks you care about.

- Task success rate for each task or workflow
- Partial completion for long-running objectives
- Correctness of the final answer against a rubric or ground truth
- Constraint satisfaction for safety, policy, and business rules
- Pass-at-k if the agent gets multiple attempts or tool paths

### 2. Tool use and action quality

Modern agents live or die by their tool use.

- Correct tool selection for each subtask
- Parameter quality for API calls and search requests
- Correct ordering and coordination across tools
- Rate of retries, dead ends, and avoidable tool errors
- Ability to recover from timeouts and partial failures

### 3. Efficiency, latency, and cost

Even a highly capable agent can be unusable if it is slow or expensive.

- Steps to completion
- End-to-end latency
- Token consumption per task
- External API usage and error rates
- Cost per successful resolution

### 4. Robustness and safety

Agents operate in open-ended environments, often with unpredictable inputs and tool responses.

- Hallucination rate
- Behavior under noisy or adversarial prompts
- Compliance with safety and policy constraints
- Recovery quality after tool failures
- Sensitivity to missing context or bad inputs

### 5. User experience and production health

A high-quality agent must also feel reliable in production.

- Human preference scores
- Perceived clarity and usefulness
- Escalation or complaint rates
- Regressions between versions
- Metric drift over time

---

## How to measure it

### Static test sets

Static evaluations are still a baseline. They make regression testing fast, repeatable, and easy to compare across versions.

### Agentic benchmarks

Benchmark suites are useful when you need realistic, multi-step tasks for browsers, desktop flows, or code agents.

### LLM-as-a-judge

LLM judges can score outputs and traces at scale when you provide a clear rubric. They are especially useful for evaluating plan quality, tool choice, and policy adherence.

### Human review

Humans are still essential for calibration, edge cases, and high-risk scenarios. A practical pattern is to use human review for the most important slices, then use those labels to validate automated judges.

### Simulation and shadow traffic

Before rollout, simulate large numbers of interactions or run the agent in shadow mode against real traffic. This helps catch rare failures before customers do.

---

## Useful tooling categories

### Evaluation and observability platforms

- LangWatch
- Maxim AI
- Arize Phoenix
- Langfuse
- Comet Opik

### Framework-integrated evaluation tools

- LangSmith for LangChain or LangGraph stacks
- DeepEval for custom LLM evaluation workflows
- Ragas for retrieval and answer quality checks

### Benchmark-style harnesses

- Browser task benchmarks
- Desktop or operating system control benchmarks
- Code agent benchmarks for multi-file changes and debugging

---

## A practical evaluation loop

### 1. Start with real tasks

Build a small but representative task set from production-like workflows.

### 2. Add trace capture

Record each step, tool call, and intermediate outcome so you can inspect failures rather than only looking at final answers.

### 3. Score the right dimensions

Measure success, cost, latency, tool quality, and policy adherence together. Optimizing only one dimension often creates regressions elsewhere.

### 4. Review failures weekly

Look for common patterns, such as routing mistakes, grounding gaps, or poor retry logic.

### 5. Gate releases

Use a prerendered or CI-friendly evaluation suite to block regressions before deployment.

### 6. Keep production in the loop

Sample live traces and run periodic offline scoring so your evaluation system evolves with your product.

---

Evaluating agents in 2026 is no longer just about measuring model accuracy on a static dataset. It is about understanding how complex, tool-using systems behave over time, across environments, and under real-world constraints.

A practical guide to evaluating AI agents across correctness, tool use, cost, robustness, simulations, observability, and production feedback loops.

Tom Vellavoor Saji

Senior AI Engineer — Azure AI, Agentic Systems, and Enterprise LLM Platforms

Latest post

Building Reliable SQL Agents in Complex, Legacy Environments

More posts

Composer 2 vs Claude & Codex: Why Cursor’s Model Is So Efficient (and What AI Engineers Can Learn)

Dreamer: The Personal Agent OS That Wants To Run Your Day

How to Evaluate AI Agent Systems in 2026: Metrics, Methods, and Tools

Side projects

AI Agent Builder Platform

LangChain RAG Agent