AI Evals and Analytics

Cohort 3

Cohort 1

Cohort 2

Cohort 2 - skipped

LLM-as-a-judge is widely used as a low-cost proxy for human or business ground truth, but uncalibrated judge scores can be statistically misleading, even reversing model rankings. This creates real production risk. Eddie Landesberg, an AI Evals researcher, introduces a calibration method to better align LLM-as-a-judge with human judgment and real-world decisions. This deck is from a guest Lightning Lesson by <a href="https://www.linkedin.com/in/eddie-landesberg/" rel="noopener noreferrer" target="_blank">Eddie Landesberg</a>. Check out Lightning Lesson the recording <a href="https://maven.com/p/ed75d8/calibrate-llm-as-a-judge-for-real-world-impact?utm_medium=ll_share_link&amp;utm_source=instructor" rel="noopener noreferrer" target="_blank">here</a>.Also check out Eddie's post "<a href="https://www.cimolabs.com/blog/metrics-lying" rel="noopener noreferrer" target="_blank">Your AI Metrics Are Lying to You</a>".

Calibrate LLM-as-a-judge for Real-world Impact

AI product teams often struggle to decide which metrics actually matter. We introduce AI evaluations and AI product analytics as two distinct but complementary approaches to measuring AI product success. We explain what AI evals measure before and during deployment, how product analytics reflects real-world behavior after launch, and how combining both across the product lifecycle helps teams build AI systems that are reliable, safe, and impactful in production.

AI Evals vs. Product Metrics

The first step in building a solid AI Evals practice is team alignment. Before you run tests or build dashboards, you need to convince leadership and stakeholders that AI Evals matter.Not everyone’s on the same page (yet). Let’s talk about how to have that conversation, and how to influence effectively.

Bring AI Evals to Product Leadership

Product-focused. Immediately executable AI Evals playbook for your product.

A Preview of Our AI Evals Playbook and Framework

Learn how to evaluate, ship, and monitor GenAI products that perform reliably in production.

Head of AI Applied Science

Stella Liu

Cofounder, AI Evals & Analytics

Amy Chen

Cofounder at AI Evals & Analytics

AI Evals and Analytics Playbook

Product Manager

Technology Advisor

Data Scientist

New Grad

Founder

AI Engineer/ AI Scientist

Staff Machine Learning Engineer

AI product manager

Senior Data Scientist

Data Scientist - Analytics

Shopify

Carvana

Harvard University

System1

Uptake

Seasalt.ai

UCLA

University of Washington

<blockquote>Building AI is easier than ever. Building AI that is reliable, safe, and trusted is not.AI evaluations help teams measure quality, detect failures, and build guardrails around your GenAI systems that will survive in the real world.</blockquote><blockquote>🔹</blockquote><h3>Who this course is for</h3>Data scientists, QA/QE professionals, and PMs who:<ul><li>have no prior background in AI evals, or</li><li>have tried LLM-as-a-judge but want a clearer framework.</li></ul><blockquote>🔹How the course works</blockquote><ul><li>Self-paced lectures you can watch anytime</li><li>Hands-on exercises using a pre-built GenAI product</li><li>Dedicated office hours for questions and guidance</li></ul><blockquote>🔹</blockquote><h3>What makes this course different</h3><ul><li>Playbook-driven approach</li><li>Evaluation-driven development</li><li>Product-focused</li><li>Hands-on experience</li><li>Flexible learning + expert support</li></ul>🔹Curious about the ideas behind this course? You can explore related topics in the <a target="_blank" rel="noopener noreferrer nofollow" href="https://datasciencexai.substack.com/">Data Science × AI Substack</a>, where we share perspectives on evaluating real GenAI products.

Learn How to Evaluate GenAI Products Confidently in 3 Weeks

Execution-ready Testing & Monitoring Playbook

GenAI Risk Management Strategy

Hands-On AI Evals Exercise

Product LeadersBuilding GenAI products and need a clear playbook for evaluation frameworks, success metrics, and shipping with confidence.

Data ScientistsLeveling up with AI product skills: learn AI evals, LLM-as-a-judge, and design metrics for GenAI systems.

QA and Test EngineerAdapting traditional testing practices to GenAI systems.

Collaborative AI Evals with Human Feedback

Rogério Chaves

CTO & Cofounder at Langwatch

Why running evals is a team sport. PMs, data scientists, and SMEs all need a seat at the table, not just the engineers.

Why collaboration is essential to AI evals

Collecting and calibrating human annotations before handing off judgment to automated evaluators at scale.

Human feedback loops before scaling LLM-as-a-judge

Create, review, & iterate on evals without needing to write code or understand technical details

Design evals and simulations for non-technical teammates

Hear customer stories from LangWatch on why cross-functional feedback is key to successful evals

Real-world customer stories from Langwatch

Evals for Voice AI: Learnings from Google Evals Team

Ravin Kumar

Sr. Researcher at Google Deepmind

Gain practical insights for building, benchmarking, or researching evals in voice and other multimodal contexts

Multimodal Evals Challenges

Learn from practical experience of evals for real-world voice-based conversational AI product like NotebookLM

Design Voice AI Evals

Use evals for real-world conversational quality, setting evals metrics and guardrails in voice systems.

Evaluation Driven Development

Eddie Landesberg

Founder of CIMO Labs

Learn why LLM-as-a-judge is a noisy biased signal rather than ground truth and how to interpret eval results accordingly

A new mental model for LLM-as-a-judge

Learn a calibration-first approach that uses limited human judgment to correct systematic errors in automated evaluators

Calibration as an AI eval design choice

Rethink low-cost eval shortcuts and design eval pipelines that better reflect real-world impact, risk, and decisions

Rethinking “cheap evals” vs. decision risk

AI Evals vs. Product Metrics: A 30min Crash Course

Head of Applied Scientist, AI Evals

Learn how to translate between business outcomes and concrete AI metrics

When and why to align AI metrics with product goals

How to use evals pre-release and analytics post-release to guide shipping, monitoring, and iteration

Evals vs. product metrics across product lifecycle

Real-world comparison of metrics for products like ChatGPT vs. your own customer service chatbot

Evals and product metrics for ChatGPT vs. for CX chatbot

How to combine product KPIs and evals dashboards into a simple, repeatable improvement system using a playbook

Build an Evals + Analytics feedback loop

Pragmatic AI Evals: Beyond Vendor Promises

Co-Founder at AI Evals & Analytics

When to consider purchasing an evals tool and how to avoid premature adoption that wastes resources

When and Why to Adopt an AI Evals Vendor

Understand how to use your internal evals playbook, team alignment, and clear metrics to drive tool selection

Design Your AI Evals Playbook First—Then Select Tools

Gain tactics for matching tool capabilities to your project's needs, from pre-release evals to post-production analytics

How to Assess Fit for Your Use Case

How to Drive AI Evals Adoption

Dr Sebastian Fox

CEO & Co-founder at Composo.ai

AI eval adoption hurdles: from cultural pushback to tough technical trade-offs.

Inside Look at Real-World Evals Challenges

Evals beyond benchmarks: capture user value, ensure consistency, and balance automation with human judgement.

Lessons from Building Evals for LLM Features & Agents

Explores limits of LLM-as-judge methods and scalable AI evaluation alternatives

Learn directly from AI Evals and Analytics

AI Evals and Analytics

Courses

AI Evals and Analytics Playbook

Lightning Lessons

Collaborative AI Evals with Human Feedback

Evals for Voice AI: Learnings from Google Evals Team

Calibrate LLM-as-a-judge for Real-world Impact

AI Evals vs. Product Metrics: A 30min Crash Course

Pragmatic AI Evals: Beyond Vendor Promises

How to Drive AI Evals Adoption

Resources

Calibrate LLM-as-a-judge for Real-world Impact

AI Evals vs. Product Metrics

Bring AI Evals to Product Leadership

A Preview of Our AI Evals Playbook and Framework