👉 These are also the kinds of things we cover in our Building AI Applications course. Our final cohort starts March 9. Here is a 25% discount code for readers. 👈

Also check out Chapter 1 of our 300 page course reader covering evaluation, testing, RAG, agents, multimodal systems, fine-tuning, production deployment, and more.

Each question below has a short, concise answer and we then provide more detail in an “In practice” section.

Builders mentioned in this post were students, builders in residence, or guest speakers in the course.

This post was co-written by Hugo Bowne-Anderson, Stefan Krawczyk, Pastor Soto, and Mike Powers. Mike and Pastor were instrumental in building the pipeline that took 1,000s of Discord messages and workshop transcripts to extract the questions (and parts of the answers!)

The top 10 questions were:

• Q1: How do I get reliable and consistent outputs from LLMs?

• Q2: What’s the difference between an LLM, augmented LLMs, LLM workflows, agents, and multi-agent systems?

• Q3: What is an Agent Harness?

• Q4: When do I use Retrieval/RAG vs Context Engineering?

• Q5: How do I choose which frameworks and tools to adopt?

• Q6: What are guardrails in LLM applications and how do I implement them effectively?

• Q7: How do I write deterministic tests for non-deterministic LLM outputs?

• Q8: What is an MVE (Minimum Viable Eval) and how do I get started with AIevaluation?

• Q9: What is an LLM Judge and how do I use them to evaluate other LLM outputs?

• Q10: When and how should I fine-tune models?

Q1: How do I get reliable and consistent outputs from LLMs?

You don’t eliminate non-determinism, you build processes around it. LLMs are fundamentally stochastic: same input, different output. The practitioners who ship reliable LLM applications have accepted this and invest in three things:

1) prompt & context engineering with tight feedback loops,

2) structured outputs and post-processing validation, and

3) systematic evaluation and testing.

In practice

Start with your prompts and context. Treat them like code: version control them, change one variable at a time, and iterate based on observed failures. Nathan Danielsen (AI Engineer, Carvana) did over 600 iterations on an initial MVP production prompt (and hundreds more after launch). Use system prompts to set behavior, delimiters to separate instructions from data, and structured output modes (JSON, Pydantic schemas) to enforce format consistency. Also look at distributions of prompt results by sending the same prompt to the system 20 times (for example).

Build evaluation and testing from day one. The highest-value activity is putting traces in a spreadsheet, labeling them pass/fail, and categorizing where things go wrong. Start with 20 representative queries. This manual inspection reveals failure modes that no automated system will catch on its own. From there, scale with code-based checks (regex, schema validation, length checks) and LLM-as-judge evaluators, always preferring deterministic checks where they work.

Design for modularity. Break large tasks into smaller, more constrained sub-tasks. One example in the course is a veterinary transcription app that improved dramatically by giving the LLM specific, smaller extraction tasks rather than one big request. This reduces variance and makes each piece independently testable. Build so you can swap models, change prompts, and add guardrails without rewriting everything.

Don’t use an LLM when you don’t need one. Regex for date extraction, fuzzy matching libraries for string similarity, etc…: these are deterministic, cheaper, and often more reliable than an LLM call for specific tasks.

The bottom line: you will never have certainty with these systems. What you can have is a process that catches most failures before they reach your users.

Important note: As William Horton (MLE, Maven Clinic) pointed out, non-determinism doesn’t matter if the result is the same. For example, a medical chatbot asked “What should I do for a mild fever?” might respond “Rest, stay hydrated, and take acetaminophen: see a doctor if it exceeds 103°F” or it might respond “An OTC pain reliever like acetaminophen can help, along with fluids and rest: consult your physician if the fever passes 103°F.” The wording is different but the medical guidance is identical. The non-determinism is cosmetic.

Q2: What’s the difference between an LLM, augmented LLMs, LLM workflows, agents, and multi-agent systems?

• An LLM is stateless: text in, text out with no memory or context (unless specified in the input);

• An augmented LLM adds memory to the system, retrieves information to ground the system (via RAG, for example), and/or adds tool calls (such as pinging an API, sending an email, or writing code to your local file system). As Anthropic points out, virtually all LLMs in production have “access to these augmented capabilities”;

• An LLM workflow is a system where LLMs and tools are orchestrated through predefined code paths: you as the developer decide what happens when;

• An Agent is a system where the LLM dynamically directs its own processes and tool usage. This takes the shape of an LLM with tools in a loop (see Q3 on Agent Harnesses);

• A Multi-agent system is a design pattern in which a primary orchestrating agentdelegates specialized, “heavy-duty” tasks to autonomous sub-agents to maintain system performance and focus.

In practice

The practical marker between LLM workflows and Agents: in an LLM workflow, the control flow is a DAG (directed acyclic graph) with predictable steps. In an agent, there’s a loop where the LLM decides what tool to call next, what to do with the result, and when to stop.

Where most value lives today. The augmented LLM and LLM workflows: an LLM with retrieval, memory, and a handful of tool calls covers the vast majority of production use cases. A meeting transcription bot that converts speech to text, summarizes, and emails minutes? People call it an agent because they get an email from it, but it’s really a workflow. If a simple tool call works, don’t build a fully-fledged agent.

When agents make sense. Agents shine for open-ended problems where you can’t predict the required steps in advance: research is the canonical example. Deep Research, Claude Code, and similar systems are genuinely agentic because the path is truly dynamic. But this comes with real costs: brittleness, complexity creep, and misaligned autonomy.

The two cultures of AI agents. In the course, we teach these two approaches: (1) build reliable software with workflows and few tool calls, strong evaluation, and predictable behavior; (2) build high-agency AI collaborators with strong human supervision. The danger zone is high agency with low supervision. Both cultures require evaluation and observability.

Q3: What is an Agent Harness?

An agent harness is the scaffolding around the LLM that manages tool execution, message history, and context. It is the infrastructure that wraps around an LLM to turn it into an agent.

In practice

Recall that an agent is a system where the LLM dynamically directs its own processes and tool usage: Agents are just LLMs with tools in an agentic loop. There is often a conversational loop with the user also.

The agent harness handles the

• Loop: prompting the model, parsing its output, executing tools, feeding results back

• Tool execution: actually running the code/commands the model asks for

• Context management: what goes in the prompt, token limits, history

• Safety/guardrails: confirmation prompts, sandboxing, disallowed actions

• State: keeping track of the conversation, files touched, etc.

And more.

Think of it like this: the LLM is the brain, the harness is everything else that lets it actually do things.

Embrace Harness Re-architecture: Teams must be willing to constantly reassess and rebuild. The popular agent Manus has been re-architected five times since March 2024, and LangChain’s Open Deep Research was rebuilt multiple times in a year to keep pace with model improvements. Even Anthropic rips out Claude Code’s agent harness as models improve!

The hello world of agent harnesses: this covers the loop, tool execution, and basic context management. What it doesn’t have: safety guardrails, token limits, persistence, or even a system prompt!

Two other interesting and powerful harnesses

The Pi Coding Agent: adds context loading AGENTS.md from multiple directories, persistent sessions you can resume and branch, and an extensibility system (skills, extensions, prompts);

OpenClaw (which builds on Pi): a persistent daemon (always-on, not invoked), chat as the interface (Telegram, WhatsApp, etc.), file-based continuity (SOUL.md, MEMORY.md, daily logs), proactive behavior (heartbeats, cron), pre-integrated tools (browser, sub-agents, device control), and the ability to message you without being prompted.

Thanks for reading Vanishing Gradients! Subscribe for free to receive new posts and support my work.

Subscribe

Q4: When do I use Retrieval/RAG vs Context Engineering?

These aren’t competing approaches: they’re tools at different levels of a system.

• Context engineering refers to “the set of strategies for curating and maintaining the optimal set of tokens (information/context) during LLM inference, including all the other information that may land there outside of the prompts.”

• RAG is one technique doing this via retrieval, that is, adding information from external sources to the context you provide the LLM.

In practice

Context engineering is the real skill. Andrej Karpathy’s definition: “the delicate art and science of filling the context window with just the right information for the next step.” We tend to agree as the term “prompt engineering” trivializes what is actually an engineering discipline. Every system, whether it uses RAG, tool calls, or hand-curated context, is doing context engineering. The question is always: what information does the model need to produce a good response for this specific step?

“Context engineering does a few things. Number one, it is the job today. If you’re looking to build a production system, that is the job: engineer your context window. Prompt engineering’s like script kitty versus context engineering’s more like software engineering.”

– Jeff Huber (Chroma, Guest Talk in the course)

Why you can’t just dump everything into long context. Context rot is real. See Jeff Huber’s research at Chroma: as input length increases, model performance degrades significantly: down to ~50% accuracy at 10,000 tokens for several simple tasks, even for frontier models that claim million-token contexts. Needle-in-a-haystack benchmarks don’t reflect real-world complexity. The practical threshold: William Horton (ML Engineer, Maven Clinic) suggests that if your entire corpus fits into ~100k tokens, just dump it in. Beyond that, retrieval remains essential. But there’s a cost dimension too: sending more tokens than necessary costs money, and running evaluations on RAG systems has caused some of the biggest cost overages William’s team has seen.

Start with Hybrid Search: Combining lexical search (like BM25) with semantic vector search is a robust default strategy. Their strengths and weaknesses are complementary, often leading to better recall and precision out of the box; see guest talk in our course from Jeff Huber (Chroma) here for more.

Agentic Retrieval/RAG: the model decides when and how to search. Here, instead of the system stuffing retrieved context into the initial prompt, the model itself determines via tool calling when to search and what queries to use. This is what people mean by “agentic RAG.” The retrieval isn’t going away: it’s moving from the application layer into the model’s decision loop. As John Berryman (Early Engineer on Github Copilot) said: “not only does the agent determine when to search, but it can determine how to search. If you do it right, the agent has a better understanding of your corpus and the user’s information needs, and it can do things like better filter through the searches, use more appropriate synonyms and search techniques, and do parallel searches that are more likely to get to the information the user wants.”

Context engineering becomes critical in multi-agentic systems: leading agentic systems like Manus and Claude Code leverage three main techniques to manage context effectively (in addition to writing memory to file):

1. Reduce: Actively shrink the context passed to the model. This can be done by compacting older tool calls (keeping only a summary) or using trajectory summarization to compress the entire history once it reaches a certain size.

2. Offload: Move information and complexity out of the prompt. This includes saving full tool results to an external file system for later reference. More profoundly, it means offloading the action space. Instead of giving an agent 100 different tools (which bloats the prompt), give it a few atomic tools like a bash terminal. This allows the agent to execute a vast range of commands without cluttering the context.

3. Isolate: Use multi-agent architectures to delegate token-heavy sub-tasks. A main agent can offload a complex job to a specialized sub-agent, which performs the work in its own isolated context and returns only a concise result.

Q5: How do I choose which frameworks and tools to adopt?

• Start with vanilla Python and direct API calls so you understand what your system actually does;

• Only adopt a framework when it solves a specific pain point you’ve already experienced: premature framework adoption is the fastest path to proof-of-concept purgatory;

• Choose frameworks that abstract away the code you don’t want to write or think about;

• Do NOT choose frameworks that abstract away what you need to see in your system (e.g. the prompt);

• Choose frameworks based on who builds them and their track record of building OSS dev tools and communities.

In practice

The proof-of-concept purgatory problem. Using frameworks prematurely will get you in POC purgatory. They’re great for demos: with a framework, you can build a RAG system demo in five lines of code. But it’s exactly this abstraction that traps teams: you can’t see what prompt was sent to the LLM, you can’t inspect the retrieval results, you can’t iterate on the pieces that matter. A question from Hugo: “How can we even think about iterating quickly on this MVP when we have no access to all the inner workings?”

Start with vanilla Python and direct API calls. This is our core position in the course, and it comes from watching the same pattern play out repeatedly: teams adopt a framework, build something quickly, ship it, and then 6-12 months later either sunset the product or rip out the framework and rebuild from API calls: this time understanding what they actually need.

When to add a framework. Only when it demonstrably improves outcomes. After you understand your system’s behavior through direct API calls, you’ll know which parts are tedious and error-prone. That’s where a framework earns its place. Make sure to adopt frameworks that abstract away the code you don’t want to write yourself. And when you do adopt tooling, prioritize observability: being able to see what your system is doing is more important than which framework orchestrates it.

Frameworks help you think, not do your job. Stefan’s framing resets the relationship: frameworks are conceptual scaffolding. They organize your thinking about what components a system needs. But you still have to control the API calls, the prompts, the evaluation. And underneath every framework, there’s an API call being made somewhere: remembering that grounds you when the hype gets thick.

The observability pattern. We see nearly all production use cases requiring some custom evaluation, logging, and tracing. The most common pattern is an observability tool (Pydantic + Logfire, Braintrust, Arize, etc.) combined with custom instrumentation. Observability tooling is being commoditized: so choose tools that integrate well and let you switch easily, rather than locking into a platform.

Q6: What are guardrails in LLM applications and how do I implement them effectively?

Guardrails are programmatic checks you run before or after an LLM call. There are two places to put guardrails:

1. Checking the input before it reaches the LLM (prompt injection detection, PII filtering, scope validation) and

2. Checking the output after (hallucination detection/groundedness, format validation).

That’s it.

In practice

What guardrails actually are. They range from simple if-else checks (is the output suspiciously short?) to regex (does the input contain a social security number?) to ML models (a fine-tuned BERT for prompt injection detection) to another LLM call (is this response faithful to the provided context?). Off-the-shelf guardrail libraries are just combinations of these same building blocks and we recommend against blindly adopting these; they are at best often too generic to provide strong value in specific use cases and can also mislead.

Start with the system prompt, then add programmatic checks. The cheapest guardrail is a well-written system prompt: “don’t make medical claims,” “cite sources,” “stay on topic.” Many production systems leak their system prompts, and they start with pages of “do not do this.” For input guardrails, delimiters (XML tags, triple backticks) help prevent prompt injection by clearly separating instructions from user data. For output guardrails, use code-based checks before reaching for an LLM judge:they’re faster, cheaper, and deterministic. In the end, you also need to be running tests (See Q7 below).

Make guardrails modular. Build them so you can rip them out and replace them. The guardrails you need will change as your product evolves, as models improve, and as you discover new failure modes.

Guardrails come in layers, and the layers have different properties. Static filters(regex, blocklists, bloom filter matching) are fast and cheap but brittle. Guest Speaker Katharine Jarmul (Privacy expert) showed how GitHub Copilot’s code guardrails (which use bloom filter-style matching to block copyrighted code) were bypassed by simply changing variable names to French. Algorithmic guardrails (classifiers like Llama Guard, LLM-as-judge evaluators) are more nuanced but add latency and cost, and have their own failure modes that need their own evaluation. And then there’s alignment (RLHF, constitutional AI) which shapes model behavior at training time. Most robust, least transparent, and not something most practitioners control directly. Effective systems layer all three: static filters catch the obvious stuff fast, model-based classifiers handle nuance, alignment sets the behavioral baseline. Katharine’s architecture:

input processing → algorithmic guardrails → LLM → algorithmic guardrails → software guardrails → user.

Real-world failure modes. ML Engineer William Horton’s team at Included Health built a customer support bot using internal documents, and it leaked direct phone numbers that were only supposed to be used by support agents. The system wasn’t malicious: it was doing exactly what it was trained to do, just with insufficient guardrails on what information should be exposed.

“Guardrails” means different things to different people. Nicola Roberts (AI Governance Lead, Australia Post) made a point that resonated across cohorts: when executives say “do we have guardrails in place?”, they’re not asking about input/output checks. They’re asking about compliance, risk appetite, accountability, and fallback plans. It’s a marketing term for organizational coverage as much as it is a technical concept.

Q7: How do I write deterministic tests for non-deterministic LLM outputs?

You DON’T write deterministic tests for non-deterministic outputs. You write tests that account for variance, you separate what can be checked deterministically with code (valid JSON, correct format, banned words absent) from what requires judgment (tone, relevance), and you track pass rates over time rather than expecting 100%.

In practice

Don’t aim for 100% pass rates. This is the hardest mindset shift for software engineers. With traditional software, tests should always pass. With LLM applications, an 85% pass rate might be excellent. The goal is to track whether that number goes up or down when you make changes. If your entire test suite passes 100% of the time, your tests aren’t hard enough: add harder cases. Put another way: think “evaluation” not “test.”

Run the same call multiple times to characterize variance. Before deciding if an output is “wrong,” understand the distribution. Stefan recommends calling the same prompt multiple times and looking at how much the output varies. Some fields (like a person’s name) should always be correct. Others (like a skills summary) will have acceptable variance.

Build curated datasets that define your product’s boundaries. Your test dataset isn’t just for testing: it defines what your product should and shouldn’t do. Start with ~20 examples for an MVE, grow to ~100 for more confidence. Use failures you encounter during development as new test cases.

Use a test runner as infrastructure, but think “evaluation” not “test.” You want parameterized cases, CI integration, and fixtures. pytest is the common choice in Python; use whatever your team knows. The key shift: your LLM checks are evaluations with acceptable failure rates, not tests that must always pass.

Two loops, one process. Your dev tests and production evals should be the same system. Don’t build a pytest suite for dev and a separate eval pipeline for prod. Same tests, same data format, same pass rate tracking. CI runs the suite before deploy; production traces feed new test cases back into development.

Run all your test cases, even when some fail. Default test behavior is to stop at the first failure, but for LLM evals you need the full picture. Run everything, collect all outputs, then analyze patterns: “name extraction passes 95%, skills summary passes 60%.” That pattern recognition beats chasing individual failures. In pytest, the pytest-harvest plugin does this; other runners have equivalents.

Q8: What is an MVE (Minimum Viable Eval) and how do I get started with AI evaluation?

An MVE (Minimum Viable Eval) is a set of input-output pairs for your AI system plus a script that runs your system’s outputs against that set. The script can include code checks (string matching, regex, structured output validation) and LLM-as-judge calls. Together, they form your eval harness: the thing that tells you whether a change made your system better or worse.

In practice

You don’t need to (and shouldn’t) jump to the most sophisticated evaluation frameworks immediately. There is a ladder of evaluation:

• Vibe checks

• Failure Analysis

• Automated evals / Eval Harness (code checks + LLM Judges)

Vibe checks

Start by looking at your data. Before you automate anything, look at 20 input-output pairs from your system. Why 20? It’s a number that seems doable and people almost always get interested when they do that: they see patterns, discover failure modes they hadn’t anticipated, and want to look at more! At this point, you can iterate quickly.

Failure Analysis

Building your MVE. From there, build up to 50-100 samples for your MVE. The experts in the space (Shreya Shankar, Hamel Husain) want at least 100 data points. Natalia Rodnova, working in healthcare, needs thousands. Nathan, AI Engineer at a Fortune 500, reviews 1,200+ samples along various dimensions for each project. But start with 50-100! Iterate as you go: what you want is representative coverage over what types of inputs your users provide.

Spreadsheets & pivot tables. Pass the input samples through your system. Label the responses by hand: accept or reject, with a reason and a failure mode classification. Put it in a spreadsheet. Do a pivot table. You now have failure modes ranked by frequency, and that tells you what to fix first. Hugo acknowledges that people hate it when he tells them to use spreadsheets and pivot tables, but it works.

Pro tip: you don’t need users to begin. Generate synthetic queries by specifying who your user is, what their goal is, and what scenario they’re in.

Automated evals

When to add automated evals & LLM judges. Start with human annotation first. Only move to automated judges after you’ve done enough manual labeling to understand your quality criteria and failure modes. Build the judge by defining clear evaluation dimensions, providing few-shot examples or heuristic rules, and then aligning the judge against your human labels. The judges aren’t a replacement for understanding your data: they’re a way to scale that understanding (more on LLM judges below in Q8).

What’s most important

The virtuous cycle. Deploy, observe behavior, make improvements, ship them, observe again. Evaluation drives this loop. One builder in the course reported going from 86.79% to 94.74% F1 score across three iterations using an eval harness. Nathan Danielsen spends 70-80% of his time on evaluations for Fortune 500 AI projects: “Changing the prompt is a very small part, compared to proving out with some good evaluations that the change to the prompt is measurable and will scale.”

“If your eyes are bleeding from spreadsheets, you’re probably doing LLM evaluation right.” — Philip Carter (Honeycomb, Guest Talk in Course)

Q9: What is an LLM Judge and how do I use them to evaluate other LLM outputs?

An LLM judge is just a prompt that scores another LLM’s output: the key is aligning it with human labels through iteration (run the judge, hand-label the same outputs blind, compare, and refine the judge prompt until they agree), and building one judge per failure mode rather than one judge for everything.

In practice

Do NOT start with judges or automated evals. Use the ladder of evaluation (see Q8 above).

• Use code-based checks for reference-based metrics, i.e. when we have a ground-truth answer (such as “was the correct tool called by an AI travel assistant?”)

• Use LLM Judges for reference-free metrics, i.e. when a “correct” response is subjective or there can multiple correct responses, such as “did the travel assistant have a helpful tone?” or “did the medical information assistant offer a diagnosis?”

The most important step in using an LLM Judge is aligning with your own judgements and/or those of an SME

Start with code-based checks. Before reaching for a judge, ask: can I check this with code? JSON schema validation, regex pattern matching, length checks, keyword matching: these are deterministic, cheap (or free), and fast. These are great for reference-based metrics (when we have a ground-truth answer), such as “was the correct tool called by an AI travel assistant?”

Use judges for subjective criteria: Use LLM judges for things that require judgment: tone, relevance, faithfulness to context, quality of reasoning. Also consider using them when there can multiple correct responses.

Align judges with humans: The alignment loop is the most important step. Once you’ve built your judge prompt: (1) run it on new outputs, (2) hand-label the same outputs yourself, blind, (3) compare human vs. judge labels, (4) iterate on the judge prompt where they disagree. Philip Carter (then Honeycomb, now Salesforce) saw alignment go from 68% to 82% to 94% after 3 iterations. One builder in the course saw judge accuracy jump from 65% to 94% after several iterations on the course project.

Pro tips

One judge per failure mode. Don’t build a single judge that evaluates everything. If your failure modes include “hallucinated facts,” “wrong tone,” and “missing key information,” build three separate judges, each with its own prompt, criteria, and examples. This mirrors how you’d assign human reviewers: you wouldn’t ask one person to evaluate everything at once.

Use pass/fail, not Likert scales. Binary evaluation is more reliable and forces clearer criteria. LLMs aren’t great at distinguishing a 3 from a 4 on a 5-point scale. Pass or fail. If it passed, move on. If it failed, categorize why.

Don’t worry about which model to use. You’ll get far more lift from the alignment process than from model selection. Use the cheapest model that works. Note there’s some evidence of self-preference bias (Gemini rates Gemini higher), but the prompt matters more than the model.

Know when you need a judge vs. when you don’t. You need a judge when: you’ve done enough manual annotation to understand your failure modes, you need to evaluate at scale, or you want continuous production monitoring. You don’t need a judge to check if JSON is valid or if a name was extracted correctly.

Q10: When and how should I fine-tune models?

Try everything else first: most production use cases are better served by prompting, retrieval, context engineering, and structured outputs. Fine-tuning still makes sense for small models on edge devices or specialized domains with poor training coverage, but it comes with real costs in infrastructure, maintenance, and reduced flexibility.

In practice

What fine-tuning actually is. It’s continuing a pre-trained model’s training on a new, task-specific dataset. The canonical example: base (completion) models don’t answer questions; ask “what is the capital of France?” and they’ll generate a list of similar questions, because that’s what their training data looks like. Instruction tuning fine-tunes on question-answer pairs, giving us the chat models we use daily. That particular kind of fine-tuning has already been done for you by the big vendors.

When it can still make sense. Small models on edge devices are the clearest use case: if you need a model running locally on a phone with low latency, you can’t use a frontier API model, and fine-tuning a smaller model for your specific task fills that gap. Specialized data domains where the training corpus has poor coverage are another case. In a guest talk in our July cohort last year, Zach Mueller (then technical lead for Hugging Face Accelerate) gave two more examples: knowledge distillation and low-resource languages.

Why not to do it. Infrastructure costs, headcount costs, and maintenance costs aren’t cheap. And when a new model comes out, do you retrain? Fine-tuning also reduces flexibility: the model may handle expected inputs well but perform worse on unexpected ones. The practical reality is that fine-tuning is costly in money, time, and ongoing maintenance.

The protocol if you do it. Get a base model. Prepare your data (collection, cleaning, labeling, curation: this is the hardest part). Choose parameter-efficient fine-tuning (LoRA is the standard approach). Train. Monitor the loss function. Test the output manually. Then evaluate formally. Frameworks like Axolotl (for beginners) and Unsloth (for limited GPU resources) handle the plumbing.

👉 These are also the kinds of things we cover in our Building AI Applications course. Our final cohort starts March 9. Here is a 25% discount code for readers. 👈

Also check out Chapter 1 of our 300 page course reader covering evaluation, testing, RAG, agents, multimodal systems, fine-tuning, production deployment, and more.

Thank you to William Horton, Nathan Danielsen, Bradley Morris, Natalia Rodnova, and John Berryman for providing feedback on drafts of this post.

Also many thanks to the 300+ builders who asked these wonderful questions in our Building AI Applications course!

Yep, correct spelling is uv, two lower-case letters.

The two most important things that uv manages for you is a project-specific virtual environment (venv), and all the project dependencies in that venv.

Let’s start creating stuff!

Install uv

If you haven’t already, install uv on your machine. Open a shell or terminal to run these commands:

• On MacOS, make sure you have Homebrew installed, then run brew install uv to install uv.

• On Linux, run curl -LsSf https://astral.sh/uv/install.sh | sh to install uv.

• On Windows, run powershell -ExecutionPolicy ByPass -c “irm https://astral.sh/uv/install.ps1 | iex” to install uv.

• (Otherwise consult the uv installation docs).

Make sure you have uv installed and available as a command line application. You should be able to open a new shell and run the following command:

$ uv --version
uv 0.9.5 (Homebrew 2025-10-21)

By the time you read this, uv will be updated many times, so you should expect to see a newer version.

Create a project with uv

Let’s create a personal Python project for you using uv.

In this example we’ll pretend you’re caled “Alice” and use that as an example. Feel free to sub Alice with your own name or whatever name you want to use for this project.

Let’s open the shell again, and create a project folder. After that let’s change into that folder and run all further commands inside that folder.

mkdir -p alice-python
cd alice-python

(On Windows use Powershell, and you can omit the -p flag to mkdir.)

We’ll uv init to initialize the project. Run uv init --help to see the available options.

$ uv init --name alice-python --python 3.13 --description “Alice’s personal Python project”
Initialized project `alice-python`

(Again, you can use your own name and description instead of Alice.)

You can inspect the contents of the alice-python directory.

$ ls -A
.git
.gitignore
.python-version
main.py
pyproject.toml
README.md

Open the folder in your favorite editor, such as Visual Studio Code, PyCharm, or Zed. Review the files that uv init has created for you:

• It created a .git folder and added a standard .gitignore file to help track your project in Git.

• It created a pyproject.toml which is the standard (not uv-specific!) config file for Python projects.

• It created a main.py which is just a dummy file to get you started. We’ll edit this later.

• It created a .python-version file which tells uv which version of Python to use.

• It created an empty README.md which you could also use to take notes during this tutorial.

venv setup

Go back from the editor to the terminal.

Run the following to create and sync the virtual environment (venv):

$ uv sync
Using CPython 3.13.3
Creating virtual environment at: .venv
Resolved 1 package in 17ms
Audited in 0.34ms

A virtual environment (venv) is a folder (inside the .venv) that contains all the dependencies of the project. The reason we use them is to isolate the dependencies of this project from the dependencies installed globally on your computer and to isolate it from the dependencies of other projects.

List the files again or take a look at the folder structure in your editor.

$ ls -A
.git
.gitignore
.python-version
.venv
alice
main.py
pyproject.toml
README.md
uv.lock

See that:

• A uv.lock file has been generated which locks (in other word: pins) all dependencies to a current version.

• A .venv folder has been created and it now contains a valid Python virtual environment. (It’s very similar to a venv you would have created with python3 -m venv .venv.)

The uv sync command does all of the following:

* If the requested Python version is not installed on your computer, it will be installed once.
* If the venv (in the .venv folder) does not exist, it will be created.
* If the uv.lock file is not present or not up to date, it will be created or updated.
* If the dependencies in the venv are not in sync lockfile, they will be installed.

The last part about syncing dependencies is the most important part, this is why the uv sync command is called uv sync. Everything else is a pre-requisite that you could do manually using other uv subcommands. (See uv python -h, uv venv -h, and uv lock -h respoectively for details.)

Syncing also means that existing dependencies will be upgraded if necessary, and extra dependencies installed in the venv but undeclared in the lockfile will be removed.

venv activation

The way Python venvs work is that you need to activate them in each shell or terminal you open. Do so with the following command:

source .venv/bin/activate

Re-run the command if you ever open a new terminal to work on this project. Your terminal should now show the name of the active venv: (alice-python).

You should be able to test your project by running either of the following commands:

$ python main.py
Hello from alice-python!
$
$ python -m main
Hello from alice-python!

Better project setup

A quite minimal Python project was created by uv, and we’ll improve it a little bit.

Did you see that the module name is just main? (python -m main?) We’ll rename it to alice, to make it unique. Again, you can use your own name, and you can even check PyPI.org to make sure you’re using a unique name that isn’t taken by anyone else.

Actually there is already a package named alice on PyPI. But it’s not related to this tutorial or anything. It’s published by a guy named Walter. Anyways...

• Inside the alice-python folder, create a subfolder named alice (or your name, but make it lowecase only, keeping it a valid Python module or variable name).

• Then move main.py inside the new alice folder.

mkdir -p alice
mv main.py alice/

Your project setup should look like this:

alice-python/
├── alice/
│   └── main.py
├── pyproject.toml
├── README.md
└── uv.lock

The reason we do this instead of just renaming main.py to alice.py is that we want to prepare for the future where there will be more modules in the alice package.

Edit your pyproject.toml to make the name match the new name you have chosen.

- name = “alice-python”
+ name = “alice”

Technically the name in the pyproject.toml is the only place where your chosen package name has to be unique and different from other PyPI.org packages if you want to publish it.

If you don’t want to publish it, it doesn’t have to be unique in any way.

Let’s assume I want to publish it but there is a package named alice already. I could choose name = “alice2025” or anything unique in the pyproject.toml‘s name field, but keep the folder name simply alice. This way, I can still publish my alice2025 package, but to use it, I can still just do import alice.

(If any other package such as the original alice package or someone else’s alice2004 package uses the alice folder name to define an alice module, then all of these Python packages are mutually incompatible with each other.)

After each change to pyproject.toml, it’s good practice to sync the lockfile and the dependencies again.

$ uv sync
Resolved 1 package in 13ms
Audited in 0.35ms

Adding dependencies

The last thing we’ll do in this tutorial is to split the code into separate modules, and add a fancy command line interface using Click and Rich.

Run the following to add these dependencies:

uv add click rich

It will print something like this, but you might get later versions:

Resolved 7 packages in 22ms
Installed 5 packages in 17ms
 + click==8.3.0
 + markdown-it-py==4.0.0
 + mdurl==0.1.2
 + pygments==2.19.2
 + rich==14.2.0

Note that this also added the top-level dependencies in your pyproject.toml:

dependencies = [
    “click>=8.3.0”,
    “rich>=14.2.0”,
]

It also locked the specific version in uv.lock.

You can also manually edit pyproject.toml to add, change, upgrade, or remove dependencies. After you make any such changes, just run uv sync again.

Creating modules

We’ll apply good software engineering princples and separate the core logic from the user interface.

We’ll create the following structure:

alice-python/
├── alice/
│   ├── main.py       [already exists]
│   ├── greetings.py  [new]
│   └── cli.py        [new]
├── pyproject.toml
├── README.md
└── uv.lock

Our core business logic is greetings, so let’s create a file alice/greetings.py with the following content:

from rich import print
from rich.text import Text

def hello(name: str):
    print(”Hello, “, Text(name, “bold red”), “! :hugging_face:”, sep=”“)

def goodbye(name: str):
    print(”Goodbye, “, Text(name, “italic blue”), “! :wave:”, sep=”“)

Next, create the command line user interface inside alice/cli.py:

from alice.greetings import hello, goodbye

import click
@click.group()
def cli():
    pass

@cli.command(name=”hello”)
@click.option(’--name’, prompt=’Your name’, help=’The person to greet.’)
def hello_cli(name: str):
    hello(name)

@cli.command(name=”bye”)
@click.option(’--name’, prompt=’Your name’, help=’The person to say goodbye to.’)
def bye_cli(name: str):
    goodbye(name)

if __name__ == “__main__”:
    cli()

You can try out the CLI by running the following:

$ python -m alice.cli hello --name Alice
Hello, Alice! 🤗

$ python -m alice.cli bye --name Alice
Goodbye, Alice! 👋

Thanks to the rich library, you’ll see emojis and colors in the output!

Custom CLI app

Instead of having to run python -m alice.cli, we want to run just a custom CLI named alice. We’ll implement this next.

Edit your pyproject.toml to add the CLI entry point and the required build backend.

  [project]
  name = “alice-python”
  version = “0.1.0”
  description = “Alice’s personal Python project”
  readme = “README.md”
  requires-python = “>=3.13”
  dependencies = [
      “click>=8.3.0”,
      “rich>=14.2.0”,
  ]
+ 
+ [project.scripts]
+ alice = “alice.cli:cli”
+ 
+ [build-system]
+ requires = [”uv_build”]
+ build-backend = “uv_build”
+ 
+ [tool.uv.build-backend]
+ namespace = true
+ module-name = “alice”
+ module-root = “”

The sync the project again to install the new CLI.

uv sync

You can now run the CLI by simply typing alice in the terminal.

$ alice
Usage: alice [OPTIONS] COMMAND [ARGS]...

Options:
  --help  Show this message and exit.

Commands:
  bye
  hello

Thanks to click, it shows the default help message on how to use this CLI. We can use it also in interactive mode, without specifying the name in the command.

$ alice hello
Your name: World
Hello, World! 🤗

Conclusion

Congratulations! You’ve created a simple Python project from scratch using uv.

We recommend checking out the uv documentation to see what else you can do with it. Happy coding!

A bit about our guest author: Hugo Bowne- Anderson advises and teaches teams building LLM-powered systems, including engineers from Netflix, Meta, and the United Nations through my course on the AI software development lifecycle. It covers everything from retrieval and evaluation to agent design and all the steps in between. Use the code MARVELOUS25 for 25% off.

In a recent Vanishing Gradients podcast, I sat down with John Berryman, an early engineer on GitHub Copilot and author of Prompt Engineering for LLMs.

We framed a practical discussion around the “Seven Deadly Sins of AI App Development,” identifying common failure modes that derail projects.

For each sin, we offer a “penance”: a clear antidote for building more robust and reliable AI systems.

You can also listen to this as a podcast:

• Spotify

• Apple

• Full notes and more episodes
  

👉 This was a guest Q&A from our July cohort of Building AI Applications for Data Scientists and Software Engineers. Enrolment is open for our next cohort (starting November 3).👈

Sin 1: Demanding 100% Accuracy

The first sin is building an AI product with the expectation that it must be 100% accurate, especially in high-stakes domains like legal or medical documentation [00:03:15]. This mindset treats a probabilistic system like deterministic software, a mismatch that leads to unworkable project requirements and potential liability issues.

The sin is requiring these AI systems to be more accurate than you would require a human to be.

The Solution/Penance:

• Reframe the problem. The goal is not to replace human judgment but to save users time.

• Design systems that make the AI’s work transparent, allowing users to verify the output and act as the final authority.

• By setting the correct expectation, that the AI is a time-saving assistant, not an infallible oracle, you can deliver value without overpromising on reliability [00:04:30].

Sin 2: Granting Agents Too Much Autonomy

This sin involves giving an agent a large, complex, closed-form task and expecting a perfect result without supervision [00:07:20]. Due to the ambiguity of language and current model limitations, the agent will likely deliver something that technically satisfies the request but violates the user’s implicit assumptions. This is particularly ineffective for tasks with a specific required solution, though it can be useful for open-ended research.

The Solution/Penance:

• Keep the user in the loop. Break down large tasks into smaller steps and build in opportunities for the agent to ask clarifying questions.

• Make the agent’s process and intermediate steps visible to the user.

• This collaborative approach narrows the solution space and ensures the final output aligns with the user’s actual intent [00:09:10].

Sin 3: Irresponsible Context Stuffing

With ever-larger context windows, the temptation is to simply “shove it all into the prompt” and perform RAG over the full content [00:11:45]. This approach degrades model accuracy, as critical information gets lost in the noise. It also increases cost and latency without a corresponding improvement in performance.

The Solution/Penance:

• Practice responsible “context engineering.”

• Instead of brute-forcing the context window, first lay out all potential information sources.

• Then, create a relevance-ranking algorithm to tier the context, ensuring only the most vital information makes it into the prompt.

• Build a template that fits within a sensible token budget, not the maximum window size, ensuring the final context is lean, readable, and effective [00:13:20].

Sin 4: Starting with Multi-Agent Complexity

A common mistake is to jump directly to a complex, orchestrated multi-agent framework for a problem that may not require it [00:17:10]. This introduces significant overhead, creates hard-to-debug handoffs between agents, and often results in a system that underperforms a simpler, single-agent alternative.

The Solution/Penance:

• Start with the simplest viable solution.

• Build a single-prompt, single-model system first to understand its failure modes and limitations.

• Only when the pain of the simple approach becomes clear should you incrementally add complexity.

• This process ensures that if a multi-agent system is truly needed, its design is informed by real-world constraints rather than assumptions [00:19:15].

Sin 5: Treating RAG as a Black Box

Many developers adopt off-the-shelf RAG frameworks, treating them as a single, monolithic component [00:20:45]. When the system fails, this black-box approach makes it nearly impossible to diagnose whether the fault lies in the data ingestion, the retrieval step, or the generation model.

The Solution/Penance:

• Build your own RAG system, and consider the individual pieces of the pipeline: indexing, query creation, retrieval, and generation.

• By isolating these components, you can debug them independently.

• If the generated answer is wrong, you can inspect the retrieved context to see if the issue is poor retrieval.

• If the context is good but the answer is bad, the problem lies with the generation prompt or model.

• This separation is key to building a reliable and maintainable system [00:22:10].

Sin 6: Overloading a Single Prompt

This sin is the smaller-scale version of giving agents too much autonomy: trying to pack too many distinct tasks into a single LLM request [00:26:25]. Asking a model to extract multiple, unrelated pieces of information or perform several judgments at once often leads to it hallucinating, missing tasks, or producing low-quality output as its attention is divided.

The Solution/Penance:

• Break the problem into smaller, atomic pieces.

• Instead of one large prompt asking for everything, use several smaller, focused prompts, which can even be run in parallel.

• For example, rather than asking a model to “find all hallucinations,” iterate through each fact and ask, “Is this statement supported by the source? True or False.”

• This trades a small amount of latency for a large gain in accuracy and reliability [00:28:10].

Sin 7: Chasing Frameworks and Logos

The AI ecosystem is an explosion of new tools, SDKs, and models, creating a temptation to constantly search for the perfect off-the-shelf solution [00:29:15]. Over-indexing on third-party frameworks too early can lead to vendor lock-in, deep and opaque stack traces, and technical debt when the ecosystem inevitably shifts.

The Solution/Penance:

• Dig in and build essential components yourself, at least initially. Understand that at its core, an LLM call is just an HTTP request.

• For domain-specific needs like evaluation or human review, a simple, custom-built tool is often more effective than a generic platform.

• Adopt frameworks mindfully, pick one that solves a clear problem, and anticipate that you will still need to write and rewrite code as your needs evolve [00:31:10].

A Summary of Penances

• Accuracy: Reframe the goal. Instead of trying to make AI 100% accurate, make it a way to augment your users and save them time.

• Oversight: When working with complex workflows, keep the user in the loop and break down large agentic tasks into smaller, supervised steps.

• Context: Avoid model distraction by selectively curating context. Rank context by relevance and select only the most relevant information.

• Complexity: Start with the simplest possible solution before adding multi-agent complexity.

• RAG: In order to facilitate easy debugging, design RAG as a transparent pipeline of indexing, query generation, retrieval, and generation.

• Prompts: Improve accuracy by decomposing complex multi-part prompts into several prompts that are easier for the model to process.

• Frameworks: Build core components yourself and adopt third-party tools mindfully, anticipating change.

Conclusion

The path to building reliable AI applications is paved with a deeper understanding of the models themselves.

John Berryman offers a powerful mental model:

empathize with the LLM by treating it as a “super intelligent, super eager, and a forgetful AI intern” [00:34:10]. This intern is brilliant but not psychic, gets distracted easily, and wakes up every morning (with every API call) with no memory of the past.

This perspective shifts the focus from finding the perfect tool to designing robust, debuggable systems that work with the model’s nature, not against it.

Want to go deeper? I teach a course on the entire AI software development lifecycle (use code MARVELOUS25 for 25% off), covering everything from retrieval and evaluation to observability, agents, and production workflows. It’s designed for engineers and teams who want to move fast without getting stuck in proof of concept purgatory.

If you’d like a taste of the full course, I’ve put together a free 10-part email series on building LLM-powered apps. It walks through practical strategies for escaping proof-of-concept purgatory: one clear, focused email at a time.

Copyrights: The article was originally published on the Vanishing Gradients blog.

That’s why we created a free, hands-on MLOps course using Databricks Free Edition. A guide we wish we’d had when starting out with MLOps on Databricks.

With 3 hours of video lessons and 10 detailed Substack lectures, this course takes you from zero to a production-ready ML pipeline. You’ll master automated deployment with Databricks Asset Bundles and GitHub Actions, and set up monitoring with drift detection to keep your models reliable over time.

To make things more fun, we built our use case using a toy dataset around Marvel characters, after all, we’re big fans of the universe.

The 3-hour course is now available as a single video on YouTube:

The 10-lecture journey

Here’s the full collection of articles so you can explore them easily:

Lecture 1: Introduction to MLOps

We start with the hard truth: most ML projects never make it to production. Why? Because moving from a 1000-line notebook to a scalable system requires a completely different mindset.

Lecture 2: Developing on Databricks

We show why notebooks are great for exploration but become bottlenecks for MLOps. They make it difficult to write modular code, apply code quality standards, or run unit tests. We’ll show you how to use VS Code extension, Databricks CLI, and Databricks Connect to develop locally with modern engineering workflows while running PySpark code on Databricks.

Lecture 3: Getting started with MLflow

Dive deep into the two most important MLflow classes: `mlflow.entities.Experiment` and `mlflow.entities.Run`. These form the foundation of everything else in MLflow. Learn how to set up tracking and registry URIs for Databricks, implement environment detection to distinguish local vs. Databricks environments, and handle authentication profiles for multiple developers collaborating on the same codebase.

Lecture 4: Logging and registering models with MLflow

Learn the difference between logging a pickle file and creating a proper MLflow Model. It’s the difference between a prototype and production-ready code.

Lecture 5: Model serving architectures

Not all ML model serving is created equal. We cover:

- Batch predictions (pre-compute, store, and retrieve on demand)

- Real-time serving (compute on demand)

- Hybrid approaches (feature lookup + real-time inference)

Lecture 6: Deploying a model serving endpoint

Deploy your first model endpoint with one Python command using Databricks SDK. Databricks Model Serving provides fully managed serverless model serving APIs with automatic scaling (including scale-to-zero), built-in monitoring for latency/throughput/error rates, and seamless Unity Catalog integration. Implement A/B testing with traffic splitting and model comparison strategies.

Lecture 7: Databricks Asset Bundles

DAB provides the recommended declarative, YAML-based approach for deploying resources and dependencies on Databricks. It offers a balance between Terraform’s complexity and API flexibility. Under the hood, it leverages Terraform while managing the complexity for you. We deploy a ML workflow using DABs.

Lecture 8: CI/CD and deployment strategies

Build proper deployment pipelines with Unity Catalog governance. Learn the difference between workspace-level and catalog-level permissions (it matters more than you think). Create CI/CD pipelines for automated testing and deployment to multiple environment.

Lecture 9: Introduction to ML monitoring

Traditional monitoring isn’t enough for ML systems. Learn the difference between data drift (input distribution shifts) and concept drift (relationship between inputs and outputs changes). Understand that not all drift is bad: sometimes models remain robust despite input changes. ML monitoring requires tracking statistical properties beyond system health, errors, and latency.

Lecture 10: Implementing Model Monitoring on Databricks

Implement end-to-end monitoring using Databricks Lakehouse Monitoring. Build four key components: inference logging (capturing model inputs/outputs), monitoring table creation, scheduled refreshes, and monitoring dashboards. Learn how Databricks automatically generates Profile Metrics Tables (summary stats, accuracy, confusion matrix) and Drift Metrics Tables for tracking data distribution changes over time.

Want to learn more, go deeper, and get personalized feedback? 1st of September we start the last cohort of our MLOps with Databricks course. If you considered to join, don’t miss this opportunity! Use this link to get 100 euro off.

We also opened enrollment for LLMOps with Databricks course starting in January 2026. Use this link to apply code EARLYBIRD for 20% off (valid until end of September).

Databricks recently introduced Free Edition, which opened the door for us to create a free hands-on course on MLOps with Databricks.

This article is part of that course series, where we walk through the tools, patterns, and best practices for building and deploying machine learning workflows on Databricks.

Watch the lecture on YouTube.

In the previous lecture, we covered the theory behind ML model monitoring and the tools Databricks provides for it. In this session, we’ll walk through a practical example, from inference tables to Lakehouse Monitoring implementation.

Our monitoring system consists of four key components:

1. Inference Logging: Capturing model inputs and outputs

2. Monitoring Table Creation: Transforming raw logs into a format suitable for monitoring

3. Scheduled Refreshes: Keeping monitoring data up-to-date

4. Monitoring Dashboard: Visualizing metrics and detecting drift

Let’s examine each component in detail.

1. Inference Data Collection

!Make sure that inference tables is enabled for your serving endpoint.

First, we need to collect data from our model serving endpoint. The notebook lecture10.marvel_create_monitoring_table.py demonstrates how to send requests to our endpoint and then process the logged data. In lecture 6, we learned how to call the model endpoint. There are two ways to do this: either via HTTPS or by using the Workspace Client.

def send_request_https(dataframe_record):
    """
    Sends a request to the model serving endpoint using HTTPS.
    """
    serving_endpoint = f"https://{os.environ['DBR_HOST']}/serving-endpoints/marvel-characters-model-serving/invocations"
    
    response = requests.post(
        serving_endpoint,
        headers={"Authorization": f"Bearer {os.environ['DBR_TOKEN']}"},
        json={"dataframe_records": dataframe_record},
    )
    return response.status_code, response.text

def send_request_workspace(dataframe_record):
    """
    Sends a request to the model serving endpoint using workspace client.
    """    
    response = workspace.serving_endpoints.query(
        name="marvel-characters-model-serving",
        dataframe_records=dataframe_record
    )
    return response

We sample records from our test set and send them to the endpoint to generate some logs.

test_set = spark.table(f"{config.catalog_name}.{config.schema_name}.test_set") \
                    .withColumn("Id", col("Id").cast("string")) \
                    .toPandas()

# Sample records for testing
sampled_records = test_set[required_columns].sample(n=100, replace=True).to_dict(orient="records")
dataframe_records = [[record] for record in sampled_records]

# Test the endpoint
for i in range(len(dataframe_records)):
    status_code, response_text = send_request_https(dataframe_records[i])
    print(f"Response Status: {status_code}")
    print(f"Response Text: {response_text}")
    time.sleep(0.2)

2. Monitoring Implementation

2.1 Creating and Refreshing Monitoring

Once logs start appearing in the inference table, we can move on to processing them into a monitoring table. For this, we’ve created a new module at src/marvel_characters/monitoring.py, which contains the logic for transforming inference logs into a structured monitoring table.

from databricks.connect import DatabricksSession
from databricks.sdk import WorkspaceClient

from marvel_characters.config import ProjectConfig
from marvel_characters.monitoring import create_or_refresh_monitoring

spark = DatabricksSession.builder.getOrCreate()
workspace = WorkspaceClient()

# Load configuration
config = ProjectConfig.from_yaml(config_path="../project_config_marvel.yml", env="dev")

create_or_refresh_monitoring(config=config, spark=spark, workspace=workspace)

This function below processes the inference data from a Delta table,
parses the request and response JSON fields, writes the resulting DataFrame to a Delta table for monitoring purposes.

def create_or_refresh_monitoring(config: ProjectConfig, spark: SparkSession, workspace: WorkspaceClient) -> None:
    """Create or refresh a monitoring table for Marvel character model serving data.

    This function processes the inference data from a Delta table,
    parses the request and response JSON fields, writes the resulting DataFrame to a Delta table for monitoring purposes.

    :param config: Configuration object containing catalog and schema names.
    :param spark: Spark session used for executing SQL queries and transformations.
    :param workspace: Workspace object used for managing quality monitors.
    """
    # Check if custom_model_payload table exists and has data
    inf_table = spark.sql(f"SELECT * FROM {config.catalog_name}.{config.schema_name}.`custom_model_payload`")
    inf_count = inf_table.count()
    logger.info(f"Found {inf_count} records in custom_model_payload table")

    if inf_count == 0:
        logger.warning("No records found in custom_model_payload table. Monitoring table will be empty.")
        return

The function defines schemas for parsing the JSON request and response data:

  request_schema = StructType(
        [
            StructField(
                "dataframe_records",
                ArrayType(
                    StructType(
                        [
                            StructField("Height", DoubleType(), True),
                            StructField("Weight", DoubleType(), True),
                            StructField("Universe", StringType(), True),
                            StructField("Identity", StringType(), True),
                            StructField("Gender", StringType(), True),
                            StructField("Marital_Status", StringType(), True),
                            StructField("Teams", StringType(), True),
                            StructField("Origin", StringType(), True),
                            StructField("Magic", StringType(), True),
                            StructField("Mutant", StringType(), True),
                        ]
                    )
                ),
                True,
            )
        ]
    )

    response_schema = StructType(
        [
            StructField("predictions", ArrayType(IntegerType()), True),
            StructField(
                "databricks_output",
                StructType(
                    [StructField("trace", StringType(), True), StructField("databricks_request_id", StringType(), True)]
                ),
                True,
            ),
        ]
    )

It then parses the JSON data and transforms it into a structured format:

 inf_table_parsed = inf_table.withColumn("parsed_request", F.from_json(F.col("request"), request_schema))
    inf_table_parsed = inf_table_parsed.withColumn("parsed_response", F.from_json(F.col("response"), response_schema))
    df_exploded = inf_table_parsed.withColumn("record", F.explode(F.col("parsed_request.dataframe_records")))

    df_final = df_exploded.withColumn("timestamp_ms", (F.col("request_time").cast("long") * 1000)).select(
        F.col("request_time").alias("timestamp"),  # Use request_time as the timestamp
        F.col("timestamp_ms"),  # Select the newly created timestamp_ms column
        "databricks_request_id",
        "execution_duration_ms",
        F.col("record.Height").alias("Height"),
        F.col("record.Weight").alias("Weight"),
        F.col("record.Universe").alias("Universe"),
        F.col("record.Identity").alias("Identity"),
        F.col("record.Gender").alias("Gender"),
        F.col("record.Marital_Status").alias("Marital_Status"),
        F.col("record.Teams").alias("Teams"),
        F.col("record.Origin").alias("Origin"),
        F.col("record.Magic").alias("Magic"),
        F.col("record.Mutant").alias("Mutant"),
        F.col("parsed_response.predictions")[0].alias("prediction"),
        F.lit("marvel-characters-model-fe").alias("model_name"),
    )

The function handles data quality issues and writes the processed data to a monitoring table:

  # Make dropna optional if we're losing all data
    df_with_valid_values = df_final_with_status.dropna(subset=["prediction"])
    valid_count = df_with_valid_values.count()
    logger.info(f"Records with valid prediction values: {valid_count}")

    # If we lost all data after dropna, use the data before dropna
    if valid_count > 0:
        df_final_with_status = df_with_valid_values
        logger.info("Using records with valid prediction values")
    else:
        logger.warning("All records have null prediction values. Using records with potential nulls.")

    # Ensure Height and Weight are properly cast to double
    df_final_with_status = df_final_with_status.withColumn("Height", F.col("Height").cast("double"))
    df_final_with_status = df_final_with_status.withColumn("Weight", F.col("Weight").cast("double"))

    # Write to the monitoring table
    df_final_with_status.write.format("delta").mode("append").saveAsTable(
        f"{config.catalog_name}.{config.schema_name}.model_monitoring"
    )

Finally, it either refreshes an existing monitoring configuration or creates a new one:

    try:
        workspace.quality_monitors.get(f"{config.catalog_name}.{config.schema_name}.model_monitoring")
        workspace.quality_monitors.run_refresh(
            table_name=f"{config.catalog_name}.{config.schema_name}.model_monitoring"
        )
        logger.info("Lakehouse monitoring table exist, refreshing.")
    except NotFound:
        create_monitoring_table(config=config, spark=spark, workspace=workspace)
        logger.info("Lakehouse monitoring table is created.")

2.2 Creating the Monitoring Table

The create_monitoring_table function sets up a new monitoring table with Databricks’ quality monitoring features:

def create_monitoring_table(config: ProjectConfig, spark: SparkSession, workspace: WorkspaceClient) -> None:
    """Create a new monitoring table for Marvel character model monitoring.

    This function sets up a monitoring table using the provided configuration,
    SparkSession, and workspace. It also enables Change Data Feed for the table.

    :param config: Configuration object containing catalog and schema names
    :param spark: SparkSession object for executing SQL commands
    :param workspace: Workspace object for creating quality monitors
    """
    logger.info("Creating new monitoring table..")

    monitoring_table = f"{config.catalog_name}.{config.schema_name}.model_monitoring"

    workspace.quality_monitors.create(
        table_name=monitoring_table,
        assets_dir=f"/Workspace/Shared/lakehouse_monitoring/{monitoring_table}",
        output_schema_name=f"{config.catalog_name}.{config.schema_name}",
        inference_log=MonitorInferenceLog(
            problem_type=MonitorInferenceLogProblemType.PROBLEM_TYPE_CLASSIFICATION,
            prediction_col="prediction",
            timestamp_col="timestamp",
            granularities=["30 minutes"],
            model_id_col="model_name",
        ),
    )

    # Important to update monitoring
    spark.sql(f"ALTER TABLE {monitoring_table} SET TBLPROPERTIES (delta.enableChangeDataFeed = true);")

    logger.info("Lakehouse monitoring table is created.")

Once lakehouse monitoring is created, you will see 2 tables automatically created under the provided catalog and schema as well as a dashboard under dashboards tab.

This dashboard has some default panels, but you can create any panel with SQL queries.

3. Scheduled Monitoring Refresh

To keep our monitoring data current, we schedule regular refreshes using Databricks Asset Bundles. The configuration is defined in resources/bundle_monitoring.yml:

resources:
  jobs:
    marvel-characters-monitor-update:
      name: marvel-characters-monitor-update-workflow
      schedule:
        quartz_cron_expression: "0 0 6 ? * MON"
        timezone_id: "Europe/Amsterdam"
        pause_status: ${var.schedule_pause_status}
      tags:
        project_name: "marvel-characters"

      environments:
        - environment_key: monitor-env
          spec:
            client: "3"
            dependencies:
              - ../dist/*.whl

      tasks:
        - task_key: "refresh_monitor_table"
          environment_key: monitor-env
          spark_python_task:
            python_file: "../scripts/refresh_monitor.py"
            parameters:
              - "--root_path"
              - ${workspace.root_path}
              - "--env"
              - ${bundle.target}

This job:

• Runs weekly on Mondays at 6:00 AM Amsterdam time

• Executes the scipts/refresh_monitor.py script with appropriate parameters

4. Monitoring Refresh Script

The scripts/refresh_monitor.py script is the entry point for our scheduled monitoring job:

import argparse
from databricks.connect import DatabricksSession
from databricks.sdk import WorkspaceClient

from marvel_characters.config import ProjectConfig
from marvel_characters.monitoring import create_or_refresh_monitoring

parser = argparse.ArgumentParser()
parser.add_argument(
    "--root_path",
    action="store",
    default=None,
    type=str,
    required=True,
)

parser.add_argument(
    "--env",
    action="store",
    default=None,
    type=str,
    required=True,
)

args = parser.parse_args()
root_path = args.root_path
config_path = f"{root_path}/files/project_config_marvel.yml"

# Load configuration
config = ProjectConfig.from_yaml(config_path=config_path, env=args.env)

spark = DatabricksSession.builder.getOrCreate()
workspace = WorkspaceClient()

create_or_refresh_monitoring(config=config, spark=spark, workspace=workspace)

This script:

1. Parses command-line arguments for the root path and environment

2. Loads the appropriate configuration

3. Calls the monitoring refresh function

Conclusion

Effective model monitoring is crucial for maintaining ML system reliability. Our implementation:

1. Captures inference data from model serving endpoints

2. Transforms this data into a monitoring-friendly format

3. Schedules regular updates to keep monitoring current

4. Uses Databricks’ built-in quality monitoring features

By implementing this monitoring system, we can detect issues early, understand model behavior in production, and make data-driven decisions about when to retrain or update our models.

This article is part of that course series, where we walk through the tools, patterns, and best practices for building and deploying machine learning workflows on Databricks.

In this lecture, we’ll dive into one of the most critical (and often misunderstood) aspects of production ML: monitoring.

Watch the lecture on Youtube:

In a ML system, you need to monitor metrics that go beyond the ones you’d expect in any production system (such as system health, errors, and latency, KPIs and infrastructure costs).

In classic software, if code, data, and environment stay the same, so does behavior. ML systems are different: model performance can degrade even if nothing changes in your code or infra because ML is driven by the statistical properties of your data. User behavior can shift, seasonality or upstream data can change.

All can cause your model to underperform, even if everything else is “the same.” That’s why MLOps monitoring includes data drift, model drift, and statistical health, not just system metrics.

Data Drift: It happens when the distribution of the input data shifts over time, even if the relationship between inputs and outputs stays the same. For example, let’s say there is a lot of new houses entering the market in a certain district. People’s preferences, the relationship between features and price stays the same.
But because the model hasn’t seen enough examples of new houses, its performance drops, not because the logic changed, but because the data shifted. In this case, data drift is the root cause of model degradation.

Concept Drift: It happens when the relationship between input features and the target variable changes over time so model’s original assumptions about how inputs relate to outputs no longer holds. Let’s look at housing prices example: new houses enter the market, and the government introduces a subsidy for families with children which leads to larger houses sold for lower prices. This is a shift in the underlying relationship between features like house size and the final price. Even if the input data distribution doesn’t change much, the model’s predictions will become less accurate

Not all drift is bad! Sometimes, your model is robust to input changes, and performance remains stable. Check this article for a real example. Suppose you detect significant data drift in the “temperature” feature using Jensen-Shannon distance. But, when you check the model’s MAE (Mean Absolute Error), performance is still well within acceptable bounds. The main point is that not all drift requires action, so we need to monitor both data and performance before retraining or raising alarms.

Let’s dive into what Databricks has to offer for ML monitoring.

ML Monitoring in Databricks

Databricks Lakehouse Monitoring lets you monitor the statistical properties and quality of the data in your delta tables. We can also use it to track the performance of machine learning models and model serving endpoints by creating inference tables with model inputs and predictions.

Databricks automatically generates a few key components to help you track model and data health.

Profile Metrics Table

• Stores summary stats for each feature in each time window (count, nulls, mean, stddev, min/max, etc.).

• For inference logs: also tracks accuracy, confusion matrix, F1, MSE, R², and fairness metrics.

• Supports slicing/grouping (e.g., by model ID or feature value).

Drift Metrics Table

• Tracks how your data’s column distributions evolve over time using advanced drift detection techniques

• Essential for identifying data quality issues, detecting shifts in real-world behavior, and ensuring that model predictions remain reliable and unbiased.

There are two primary types of drift detection:

Consecutive Drift. Compares the current time window to the previous one to detect short-term anomalies or sudden changes.
Baseline Drift. Compares the current data to a fixed reference or baseline table, typically built from training data or a known “good” state.

Drift is calculated using a combination of statistical tests, distance metrics, and simple delta metrics. The table below shows which method work for which data type.

For detailed guidelines, you can check these articles: 1, 2. Also, the Little Book of Metrics is a great resource on ML model metrics.

Inference Tables

Inference tables are a built-in feature to log model inputs and predictions from a serving endpoint directly into a Delta table in Unity Catalog. This provides a simple way to monitor, debug, and optimize models in production.

Once enabled, they automatically capture request and response payloads, as well as metadata like response time and status codes etc. It’s used for:

• Monitoring quality

• Debugging

• Training corpus generation

In order to create the inference table, you need to check enable inference table box when editing a serving endpoint (can also be created programmatically). The workspace must have Unity Catalog enabled, and you’ll need the right permissions to create and manage the associated Delta table.

Inference tables log raw data. To monitor drift/performance, process them into a structured inference profile table (with timestamp, features, prediction, and optionally ground truth).

You can set up an automated job that processes the raw inference table into a structured monitoring table. This monitoring table would be used to

• Update generate a metrics table

• Automatically update the dashboard

You can also configure alerts, so you’re notified when performance drops or data shifts.

Monitoring pipeline

Let’s take a look at a model serving use case, similar to what we cover in the course. For example, there is a dynamic pricing model that gets retrained weekly, and the updated model is deployed to the serving endpoint. The workflow is similar to what we covered in Lecture 7.

A separate workflow would be required to update the Lakehouse monitor when ground truth labels arrive. We’ll go through it in our final lecture.

In the end, we’ll have two workflows, one for training and deployment, and the other one — for monitoring:

Conclusions

Here are some key takeaways from this lecture:

• ML monitoring goes beyond system metrics, it includes data quality, drift, and model performance.

• Databricks Lakehouse Monitoring provides built-in tools for tracking data and model health over time.

• Not all data drift is bad — always check model performance before reacting.

• Inference tables + monitoring pipelines enable end-to-end visibility and alerting in production.

In the next lecture, we’ll demonstrate how to create a Lakehouse monitor on Databricks, and deploy the monitoring pipeline.

This article is part of that course series, where we walk through the tools, patterns, and best practices for building and deploying machine learning workflows on Databricks.

Watch lecture 8 on Youtube:

In this lecture, we’ll explore how to structure your data and assets for robust, secure, and scalable machine learning operations on Databricks, and how to automate deployments using CI/CD pipelines.

Unity Catalog, Workspaces, and Data Organization

We’ve already interacted with Unity Catalog, using it to create delta tables and register models. For a workspace to use Unity Catalog, it must be attached to a Unity Catalog metastore, which is the top-level container for all data and AI asset metadata.

You can only have one metastore per cloud region, and each workspace can only be attached to one metastore in that region.

Unity Catalog organizes assets in a three-tier hierarchy:

• Catalogs (e.g., mlops_dev, mlops_acc, mlops_prd)

• Schemas within catalogs (in our case, we have the same schema name in each catalog, marvel_characters)

• Assets within schemas (tables, views, models, etc.)

Assets are referenced using a three-part naming convention: catalog.schema.asset.

Access Control: Securables and Permissions

In Databricks, permissions can be set on the workspace and on Unity Catalog level.

Workspace-level securables: Notebooks, clusters, jobs — accessed via ACLs.

Unity Catalog-level securables: Tables, schemas, models — accessed via metastore-level privileges.

Workspace binding and access modes: if the catalog has OPEN mode, it can be accessed from any workspace. Use ISOLATED mode to control cross-project access.

In a typical setup, an ML project or team has a set of Databricks workspaces (dev, acc, and prd), and set of catalogs or schemas within a larger catalog.

In the course example, for simplicity we use a shared workspace for development, acceptance and production. But we have a dedicated catalog for dev, acc, and prd, with limited permissions, so that we can ensure things do not get broken unintentionally.

For an ideal setup, you need to follow these rules:

• All ML pipelines form all workspaces (dev, acc, prd) have read access to production data (e.g., prd_gold), ensuring consistency.

• From each workspace, the data can only be wtitten to its own catalog.

• Users only have direct access to the dev workspace; deployments to acc/prd must go through CI/CD pipelines, using service principals for security and traceability.

In the example below, the hotel booking team and dev SPN have “write” permissions to mlops_dev.hotel_booking, and “read” permissions on prd_gold.hotel_booking (production data delivered by the data engineering team), and the hotel_booking schema from mlops_acc and mlops_prd catalogs.

Service principals (SPNs) have scoped access, only operating within their intended workspace/catalog

Branching and Release Strategy

In the course, we use a version of Git Flow:

• Feature branches are created from main.

• Developers open PRs to main, triggering the CI pipeline.

• CI runs pre-commit checks, unit tests, and version checks.

• At least 2 approvals are required to merge (enforced via branch protection rules).

• Direct pushes to main are not allowed.

Once merged, the CD pipeline deploys to acceptance and production, using environment-scoped secrets and SPNs. Production deployment should be protected by deployment protection rules (only deployed after approval).

CI/CD in Action

Let’s look at how this is implemented in our Marvelous MLOps codebase.

CI Pipeline: .github/workflows/ci.yml : This pipeline is triggered on every push or PR to main or dev:

name: CI

on:
  pull_request:
    branches:
      - main

jobs:
  pytest_and_checks:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 #v4.2.2
        with:
          # Fetch all history for all branches and tags
          fetch-depth: 0
          ref: ${{ github.head_ref }}

      - name: Git tag from version.txt
        run: |
          echo "VERSION=$(cat version.txt)"
          git tag $VERSION

      - name: Install uv
        uses: astral-sh/setup-uv@0c5e2b8115b80b4c7c5ddf6ffdd634974642d182 #v5.4.1

      - name: Install the dependencies
        run: |
          uv sync --extra test

      - name: Run pre-commit checks
        run: |
          uv run pre-commit run --all-files

      - name: run pytest
        run: |
          uv run pytest -m "not ci_exclude"

What it does:

• Runs on PRs and pushes to main/dev

• Installs dependencies, runs linting and tests

• Checks that the version is unique (to prevent accidental duplicate releases)

CD Pipeline: .github/workflows/cd.yml

This pipeline is triggered after a successful merge to main:

name: CD

on:
  workflow_dispatch:

  push:
    branches:
        - 'main'

jobs:
  deploy:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        environment: [acc, prd]
    environment: ${{ matrix.environment }}
    permissions:
      contents: write # to push tag
    env:
      DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
      DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
      DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}
    steps:
      - name: Checkout Source Code
        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 #v4.2.2

      - name: Install Databricks CLI
        uses: databricks/setup-cli@49580195afe1ccb06d195764a1d0ae9fabfe2edd #v0.246.0
        with:
          version: 0.246.0

      - name: Configure Databricks CLI
        run: |
          mkdir -p ~/.databricks
          cat > ~/.databrickscfg << EOF
          [marvelous]
          host = ${{ vars.DATABRICKS_HOST }}
          client_id = ${{ secrets.DATABRICKS_CLIENT_ID }}
          client_secret = ${{ secrets.DATABRICKS_CLIENT_SECRET }}
          EOF

      - name: Install uv
        uses: astral-sh/setup-uv@0c5e2b8115b80b4c7c5ddf6ffdd634974642d182 #v5.4.1

      - name: Deploy to Databricks
        env:
          DATABRICKS_BUNDLE_ENV: ${{ matrix.environment }}
        run: |
            databricks bundle deploy \
              --var="git_sha=${{ github.sha }}" \
              --var="branch=${{ github.ref_name }}"
            if [ "${{ matrix.environment }}" = "prd" ]; then
            echo "VERSION=$(cat version.txt)"
            git tag $VERSION
            git push origin $VERSION
            fi

What it does:

• Only runs on push to main

• Builds the wheel (databricks bundle deploy takes care of that)

• Deploys the Lakeflow job to both acceptance and production using environment-specific secrets

• Uses the Databricks CLI to deploy bundles

Setting Up Service Principals (SPNs) for Secure CI/CD

For your CD pipeline to deploy to Databricks automatically and securely, you must use a Service Principal (SPN). This ensures that all deployments are performed by a dedicated identity with tightly scoped permissions, rather than by personal user credentials.

A Service Principal is a special, non-human identity used by applications, automation tools, or CI/CD pipelines to authenticate and interact with cloud resources securely.

Why use an SPN instead of a user account?

• Security: SPNs are not tied to any individual, so if someone leaves the team, you don’t risk losing access or exposing credentials.

• Least privilege: SPNs can be granted only the permissions they need for deployment — nothing more.

• Auditability: All actions performed by the CI/CD pipeline are clearly attributable to the SPN, making it easy to track changes and meet compliance requirements.

• Automation: SPNs enable fully automated, hands-off deployments, since their credentials (client ID and secret) can be securely stored in your CI/CD system (like GitHub Actions).

Configuring SPNs

1. Create a Service Principal in Databricks:

• Go to the Databricks workspace admin console.

• Navigate to User Management → Service Principals.

• Click Add Service Principal and follow the prompts.

• Note the Client ID and Client Secret that are generated.

Assign Permissions to the SPN:

• Grant the SPN the necessary privileges in Unity Catalog, jobs, and workspace resources (e.g., CAN_MANAGE or CAN_RUN on the relevant schemas, jobs, and endpoints).

• Make sure the SPN has access only to the environments it should deploy to (e.g., acc and prd).

2. Add SPN Credentials to GitHub Actions (or your CI/CD system)

In your GitHub repository, go to Settings → Environments. Create environments “prd” and “acc”.

Add the following Environment secrets:

• DATABRICKS_CLIENT_ID (the SPN’s client ID)

• DATABRICKS_CLIENT_SECRET (the SPN’s client secret)

Add the following Environment variable:

• DATABRICKS_HOST (your Databricks workspace URL)

These secrets and variables will be available to your GitHub Actions workflows and can be referenced as environment variables.

In your .github/workflows/cd.yml, we are referencing these secrets and variable in the env block for deployment steps. We also set the environment using matrix.

strategy:
      matrix:
        environment: [acc, prd]
environment: ${{ matrix.environment }}
env:
  DATABRICKS_HOST: ${{ vars.DATABRICKS_HOST }}
  DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
  DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}

The Databricks CLI and bundle deployment commands will pick up these variables and authenticate as the SPN.

Bonus: Authenticating to Serverless Endpoints with Service Principals

In Lecture 6, we showed how to send requests to a Databricks Serverless endpoint using a Personal Access Token (PAT).

However, for production systems and service-to-service communication, Service Principals (SPNs) provide a more secure and scalable authentication method.
Step 1: Grant Permissions to Your SPN

Before making requests, ensure your Service Principal has CAN_QUERY permission on the model serving endpoint:

1. Navigate to your endpoint in the Databricks UI

2. Click on “Permissions”

3. Add your Service Principal with “Can Query” permission

Step 2: Generate an OAuth Token Using SPN Credentials

import os
import requests

from requests.auth import HTTPBasicAuth

def get_token():
   response = requests.post(
       f"https://{os.environ['DBR_HOST']}/oidc/v1/token",
       auth=HTTPBasicAuth(
           os.environ["DATABRICKS_CLIENT_ID"],
           os.environ["DATABRICKS_CLIENT_SECRET"]
       ),
       data={
           'grant_type': 'client_credentials',
           'scope': 'all-apis'
       }
   )
   return response.json()["access_token"]

os.environ['DBR_TOKEN'] = get_token()

Step 3: Use the Token to Call Your Model Endpoint

Now you can use this token with the same endpoint invocation code from Lecture 6, replacing the authentication method:

def call_endpoint(record):
    """
    Calls the model serving endpoint with a given input record.
    """
    # Ensure the host URL is complete with domain suffix (.com, etc.)
    host = os.environ['DBR_HOST']
    # If the host doesn't contain a dot, it's likely missing the domain suffix
    if '.' not in host:
        print(f"Warning: DBR_HOST '{host}' may be incomplete. Adding '.com' domain suffix.")
        host = f"{host}.com"
        
    serving_endpoint = f"https://{host}/serving-endpoints/marvel-character-model-serving/invocations"
    
    print(f"Calling endpoint: {serving_endpoint}")
    
    response = requests.post(
        serving_endpoint,
        headers={"Authorization": f"Bearer {os.environ['DBR_TOKEN']}"},
        json={"dataframe_records": record},
    )
    return response.status_code, response.text

Key Takeaways

In this lecture, we discussed and showed how:

• Catalogs, schemas, and workspaces provide clean separation and access control.

• Service principals ensure automation is secure and scoped.

• Git flow and branch protection rules enforce code quality and review.

• CI/CD pipelines automate validation and deployment, with no manual pushes to production.

In the next lecture, we’ll talk about monitoring. Because machine learning models only start living once they are deployed, and there is no MLOps without proper monitoring.

This article is part of that course series, where we walk through the tools, patterns, and best practices for building and deploying machine learning workflows on Databricks.

In this lecture, we’ll focus on how to automate and the entire ML workflow using DAB. You can also follow along with the full walkthrough on the Marvelous MLOps YouTube channel:

All code covered in this repository is available here.

Why Databricks Asset Bundles?

When deploying resources and their dependencies on Databricks, you have a few options:

• Terraform: Full infrastructure-as-code control, but can be complex.

• Databricks APIs: Flexible, but requires custom scripting.

• Databricks Asset Bundles (DAB): The recommended, declarative, YAML-based approach.

DAB offers a balance between simplicity and power. Under the hood, it leverages Terraform, so you get all the benefits of infrastructure-as-code, without having to manage raw Terraform code yourself. This is ideal for teams looking to standardize and automate job deployments in a scalable, maintainable way.

What is DAB?

Databricks Asset Bundle (DAB) is the way to package your code, jobs, configuration, and dependencies together in a structured, version-controlled format. With DAB, you define jobs, notebooks, models, and their dependencies using YAML files.

Key features:

• Declarative YAML configuration: Define everything in one place.

• Multi-environment support: Easily target dev, staging, prod, etc.

• CI/CD friendly: Fits naturally into automated pipelines.

• Version-controlled: All changes are tracked in your repo.

What is a Lekeflow job?

Lakeflow Jobs (previously Databricks workflows) provide the execution and orchestration layer. Workflows let you run tasks (notebooks, scripts, SQL) on a schedule or in response to events, with support for dependencies, retries, parameter passing, and alerts.

Machine learning pipeline

Here’s the workflow we’ll create with DAB:

1. Preprocessing. Runs a data processing script scripts/process_data.py.

2. Train & Evaluate. Trains and evaluates the model using scripts/train_register_custom_model.py.

3. Model Update. Conditional step: if the new model is better, release a flag and register it.

4. Deployment. Deploy the registered model by creating or updating a serving endpoint using scripts/deploy_model.py.

The scripts we are running in each step can be found in the scripts folder in our repo. Parameters are passed between steps, and are explicitly defined in our databricks.yml file.

databricks.yml file

The bundle configuration is defined in the databricks.yml. The minimal configuration only contains the bundle name and the target. Let’s look at the databricks.yml file from our project:

bundle:
  name: marvel-characters

include:
  - resources/*

artifacts:
  default:
    type: whl
    build: uv build
    path: .

variables:
  git_sha:
    description: git_sha
    default: abcd
  branch:
    description: branch
    default: main
  schedule_pause_status:
    description: schedule pause status
    default: PAUSED


targets:
  dev:
    default: true
    mode: development
    workspace:
      host: <your host>
      root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.target}/${bundle.name}
      profile: marvelous
    variables:
      schedule_pause_status: PAUSED

  acc:
    presets:
      name_prefix: 'acc_'
    workspace:
      host: <your host>
      root_path: /Shared/.bundle/${bundle.target}/${bundle.name}
      profile: marvelous
    variables:
      schedule_pause_status: PAUSED

  prd:
    mode: production
    workspace:
      host: <your host>
      root_path: /Shared/.bundle/${bundle.target}/${bundle.name}
      profile: marvelous
    variables:
      schedule_pause_status: PAUSED # normally UNPAUSED

The databricks.yml file here does not contain any resources. All the resources are defined under the resource folder as separate .yml files. The resource configurations are included in the deployment thanks to the include section in the main configuration file.

The artifacts section defines how the code is packaged. The packaged code is referenced in the resources.

The variables section defines shared variables (git SHA, branch, schedule status), which can be defined per target or passed to the deployment at the deploy step.

The targets section defines deployment targets, which refer to separate environments (dev, acc, prd), each with its own settings.

Machine learning pipeline: model_deployment.yml

The resources/model_deployment.yml file defines the actual ML workflow as a Databricks job, with tasks and dependencies.

The file contains one job with a schedule that will be paused based on the variable value schedule_pause_status. We want to schedule the job in production, but pause it for acceptance target (which does not have development mode, hence will not be paused by default).

The job will run using serverless environment 3, which we discussed earlier in lecture 2.

resources:
  jobs:
    deployment:
      name: ${bundle.name}-workflow
      schedule:
        quartz_cron_expression: "0 0 6 ? * MON"
        timezone_id: "Europe/Amsterdam"
        pause_status: ${var.schedule_pause_status}
      tags:
        project_name: "marvel-characters"

      environments:
        - environment_key: default
          spec:
            client: "3"
            dependencies:
              - ../dist/*.whl

      tasks:
        - task_key: "preprocessing"
          environment_key: default
          spark_python_task:
            python_file: "../scripts/process_data.py"
            parameters:
              - "--root_path"
              - "${workspace.root_path}"
              - "--env"
              - "${bundle.target}"

        - task_key: "train_model"
          environment_key: default
          depends_on:
            - task_key: "preprocessing"
          spark_python_task:
            python_file: "../scripts/train_register_custom_model.py"
            parameters:
              - "--root_path"
              - "${workspace.root_path}"
              - "--env"
              - "${bundle.target}"
              - "--git_sha"
              - "${var.git_sha}"
              - "--job_run_id"
              - "{{job.run_id}}"
              - "--branch"
              - "${var.branch}"

        - task_key: model_updated
          condition_task:
            op: "EQUAL_TO"
            left: "{{tasks.train_model.values.model_updated}}"
            right: "1"
          depends_on:
            - task_key: "train_model"

        - task_key: "deploy_model"
          environment_key: default
          depends_on:
            - task_key: "model_updated"
              outcome: "true"
          spark_python_task:
            python_file: "../scripts/deploy_model.py"
            parameters:
              - "--root_path"
              - "${workspace.root_path}"
              - "--env"
              - "${bundle.target}"

The job consists of tasks: preprocessing, training, model update check, deployment. We pass parameters between the tasks using DABs, and also pass specific Python parameters to the tasks, such as the job run id, and git_sha, which allows us to tag MLflow runs and registered models with the actual code version.

Let’s take a closer look at each of the tasks.

Preprocessing Script: scripts/process_data.py

The preprocessing script handles data loading and preprocessing, similar to what we have done in lecture 2. We are using DataProcessor class from our custom package, which can be found under src folder.

import argparse
import yaml
from loguru import logger
from pyspark.sql import SparkSession
import pandas as pd

from marvel_characters.config import ProjectConfig
from marvel_characters.data_processor import DataProcessor

parser = argparse.ArgumentParser()
parser.add_argument(
    "--root_path",
    action="store",
    default=None,
    type=str,
    required=True,
)

parser.add_argument(
    "--env",
    action="store",
    default=None,
    type=str,
    required=True,
)


args = parser.parse_args()
config_path = f"{args.root_path}/files/project_config_marvel.yml"

config = ProjectConfig.from_yaml(config_path=config_path, env=args.env)

logger.info("Configuration loaded:")
logger.info(yaml.dump(config, default_flow_style=False))

# Load the Marvel characters dataset
spark = SparkSession.builder.getOrCreate()

# Example: Adjust the path and loading logic as per your Marvel dataset location
filepath = f"{args.root_path}/files/data/marvel_characters_dataset.csv"

# Load the data
df = pd.read_csv(filepath)

# If you have Marvel-specific synthetic/test data generation, use them here.
# Otherwise, just use the loaded Marvel dataset as is.
logger.info("Marvel data loaded for processing.")

# Initialize DataProcessor
data_processor = DataProcessor(df, config, spark)

# Preprocess the data
data_processor.preprocess()

# Split the data
X_train, X_test = data_processor.split_data()
logger.info("Training set shape: %s", X_train.shape)
logger.info("Test set shape: %s", X_test.shape)

# Save to catalog
logger.info("Saving data to catalog")
data_processor.save_to_catalog(X_train, X_test)

Training & Registration Script: scripts/train_register_custom_model.py

This script trains, evaluates, and registers the model, if the model is improved. We are using custom model class which we created in lecture 4.

import argparse

import mlflow
from loguru import logger
from pyspark.dbutils import DBUtils
from pyspark.sql import SparkSession
from importlib.metadata import version

from marvel_characters.config import ProjectConfig, Tags
from marvel_characters.models.basic_model import BasicModel
from marvel_characters.models.custom_model import MarvelModelWrapper

parser = argparse.ArgumentParser()
parser.add_argument(
    "--root_path",
    action="store",
    default=None,
    type=str,
    required=True,
)

parser.add_argument(
    "--env",
    action="store",
    default=None,
    type=str,
    required=True,
)
parser.add_argument("--git_sha", type=str, required=True, help="git sha of the commit")
parser.add_argument("--job_run_id", type=str, required=True, help="run id of the run of the databricks job")
parser.add_argument("--branch", type=str, required=True, help="branch of the project")

args = parser.parse_args()
root_path = args.root_path
config_path = f"{root_path}/files/project_config_marvel.yml"

config = ProjectConfig.from_yaml(config_path=config_path, env=args.env)
spark = SparkSession.builder.getOrCreate()
dbutils = DBUtils(spark)
tags_dict = {"git_sha": args.git_sha, "branch": args.branch, "job_run_id": args.job_run_id}
tags = Tags(**tags_dict)

# Initialize Marvel custom model
basic_model = BasicModel(config=config, tags=tags, spark=spark)
logger.info("Marvel BasicModel initialized.")

# Load Marvel data
basic_model.load_data()
logger.info("Marvel data loaded.")

# Prepare features
basic_model.prepare_features()

# Train the Marvel model
basic_model.train()
logger.info("Marvel model training completed.")

# Train the Marvel model
basic_model.log_model()

# Evaluate Marvel model
model_improved = basic_model.model_improved()
logger.info("Marvel model evaluation completed, model improved: %s", model_improved)

if model_improved:
    # Register the model
    basic_model.register_model()
    marvel_characters_v = version("marvel_characters")

    pyfunc_model_name = f"{config.catalog_name}.{config.schema_name}.marvel_character_model_custom"
    code_paths=[f"{root_path}/artifacts/.internal/marvel_characters-{marvel_characters_v}-py3-none-any.whl"]

    wrapper = MarvelModelWrapper()
    latest_version = wrapper.log_register_model(wrapped_model_uri=f"{basic_model.model_info.model_uri}",
                            pyfunc_model_name=pyfunc_model_name,
                            experiment_name=config.experiment_name_custom,
                            input_example=basic_model.X_test[0:1],
                            tags=tags,
                            code_paths=code_paths)

    logger.info("New model registered with version:", latest_version)
    dbutils.jobs.taskValues.set(key="model_version", value=latest_version)
    dbutils.jobs.taskValues.set(key="model_updated", value=1)

else:
    dbutils.jobs.taskValues.set(key="model_updated", value=0)

Deployment Script: scripts/deploy_model.py

The deployment script deploys the latest registered model to a serving endpoint. We covered the logics in lecture 6.

import argparse

from loguru import logger
from pyspark.dbutils import DBUtils
from pyspark.sql import SparkSession

from marvel_characters.config import ProjectConfig
from marvel_characters.serving.model_serving import ModelServing

parser = argparse.ArgumentParser()
parser.add_argument(
    "--root_path",
    action="store",
    default=None,
    type=str,
    required=True,
)

parser.add_argument(
    "--env",
    action="store",
    default=None,
    type=str,
    required=True,
)

args = parser.parse_args()

config_path = f"{args.root_path}/files/project_config_marvel.yml"

spark = SparkSession.builder.getOrCreate()
dbutils = DBUtils(spark)
model_version = dbutils.jobs.taskValues.get(taskKey="train_model", key="model_version")

# Load project config
config = ProjectConfig.from_yaml(config_path=config_path, env=args.env)
logger.info("Loaded config file.")

catalog_name = config.catalog_name
schema_name = config.schema_name
endpoint_name = f"marvel-characters-model-serving-{args.env}"
endpoint_name = "marvel-character-model-serving"

# Initialize Marvel Model Serving Manager
model_serving = ModelServing(
    model_name=f"{catalog_name}.{schema_name}.marvel_character_model_custom",
    endpoint_name=endpoint_name
)

# Deploy the Marvel model serving endpoint
model_serving.deploy_or_update_serving_endpoint(version=model_version)
logger.info("Started deployment/update of the Marvel serving endpoint.")

Running and Managing Bundles

We use Databricks CLI commands to validate, deploy and run our workflow.

• databricks bundle validate — validate your bundle

• databricks bundle deploy — deploys your bundle

• databricks bundle run — run the job

• databricks bundle destroy — Tear down resources

Once we deploy our bundle, we see our workflow created in the target workspace.

Conclusion

With Databricks Asset Bundles, we can package all logic, dependencies, and configurations together, define robust multi‑step ML workflows, and version and automate deployments across environments. This enables true reproducibility and aligns with CI/CD best practices, making deployments consistent and reliable.

In the next lecture, we will explore how to deploy a bundle through a CI/CD pipeline to fully automate the path from development to production.

This article is part of that course series, where we walk through the tools, patterns, and best practices for building and deploying machine learning workflows on Databricks.

This is lecture 6 (out of 10). Let’s dive into deploying model serving endpoints and implementing A/B testing on Databricks. You can also follow along with the full walkthrough on the Marvelous MLOps YouTube channel.

In previous lectures, you learned how to train, log, and register models with MLflow. Now, it’s time to expose those models behind an API using Databricks Model Serving.

Databricks Model Serving is a fully managed, serverless solution that allows you to deploy MLflow models as RESTful APIs without the need to set up or manage any infrastructure.

• Effortless deployment of registered MLflow models

• Automatic scaling, including scale-to-zero when there’s no traffic

• Built-in monitoring in the Databricks UI (track latency, throughput, error rates)

• Seamless integration with models registered in Unity Catalog

Model serving limitations

Databricks model serving makes the transition from experimentation to production incredibly smooth. It’s ideal for teams who want to focus on building great models, not managing infrastructure. However, if you choose to deploy a model serving endpoint on Databricks, you must be aware of its limitations, such as:

• No control over runtime environment: Databricks chooses the environment for you, which can be a constraint if you need specific library versions.

• No control over cluster size. Each replica is limited to 4 GB RAM (CPU), which may not be enough for very large models.

• Workload size options: You can choose the workload size (Small, Medium, Large, XL, etc.), which determines the number of compute units per replica. For demanding use cases, you can scale up to 512 units per endpoint on request.

The workload size determines the number of compute units available, with each unit able to handle one request at a time (4 for Small, 8–16 for Medium, 16–64 for Large). This does not directly translate to queries per second (QPS), as throughput depends on the execution time of the model’s predict function. For example, if prediction takes 20 ms, 4 compute units can handle approximately 4 / 0.02 = 200 QPS.

Autoscaling is based on the number of required units, not CPU or RAM usage. The number of required units is calculated as:

• Required units = Queries per second × Model Processing Time (in seconds).

Example: 1,000 QPS × 0.02s = 20 units needed.

Payload structure

Another serving limitation is the payload structure. Databricks uses MLflow serving behind the scenes, and the payload is defined by it. You have some ability to adapt the payload for your needs by using pyfunc, but you have no influence on the global payload structure.

This is not ideal if you already have existing integrations, and want to migrate to Databricks model serving.

Model deployment

Let’s go through notebooks/lecture6.deploy_model_serving_endpoint.py notebook. You can find the code repository here.

Before we proceed with the model deployment, we need to authenticate with Databricks and set up our environment (the details on authentication are covered in lecture 2). The code supports both running inside Databricks and running locally (e.g., from VS Code). Authentication details are loaded from environment variables or the .env file, ensuring a smooth developer experience for teams.

We also define the DBR_TOKEN and DBR_HOST environment variables, which will be used later to send requests to the deployed endpoint.

from databricks.sdk import WorkspaceClient
import os
from dotenv import load_dotenv
import mlflow
from marvel_characters.utils import is_databricks

w = WorkspaceClient()
os.environ["DBR_HOST"] = w.config.host
os.environ["DBR_TOKEN"] = w.tokens.create(lifetime_seconds=1200).token_value

if not is_databricks():
    load_dotenv()
    profile = os.environ["PROFILE"]
    mlflow.set_tracking_uri(f"databricks://{profile}")
    mlflow.set_registry_uri(f"databricks-uc://{profile}")

The deployment leverages a custom ModelServing utility that wraps Databricks Model Serving APIs. Here’s how you deploy or update the endpoint:

from marvel_characters.serving.model_serving import ModelServing

model_serving = ModelServing(
    model_name=f"{catalog_name}.{schema_name}.marvel_character_model_custom",
    endpoint_name="marvel-character-model-serving"
)

model_serving.deploy_or_update_serving_endpoint()

Let’s take a look how the ModelServing class and the deploy_or_update_serving_endpoint() method look like. Essentially, it is a wrapper around Databricks sdk, where we take the latest registered model by the latest-model alias, and use it to create or update the endpoint if it already exists.

import mlflow
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.serving import (
    EndpointCoreConfigInput,
    ServedEntityInput,
)


class ModelServing:
    """Manages model serving in Databricks for Marvel characters."""

    def __init__(self, model_name: str, endpoint_name: str) -> None:
        """Initialize the Model Serving Manager.
        """
        self.workspace = WorkspaceClient()
        self.endpoint_name = endpoint_name
        self.model_name = model_name

    def get_latest_model_version(self) -> str:
        """Retrieve the latest version of the model.
        """
        client = mlflow.MlflowClient()
        latest_version = client.get_model_version_by_alias(self.model_name, alias="latest-model").version
        print(f"Latest model version: {latest_version}")
        return latest_version
    def deploy_or_update_serving_endpoint(
        self, version: str = "latest", workload_size: str = "Small", scale_to_zero: bool = True
    ) -> None:
        """Deploy or update the model serving endpoint in Databricks for Marvel characters.
        """
        endpoint_exists = any(item.name == self.endpoint_name for item in self.workspace.serving_endpoints.list())
        entity_version = self.get_latest_model_version() if version == "latest" else version

        served_entities = [
            ServedEntityInput(
                entity_name=self.model_name,
                scale_to_zero_enabled=scale_to_zero,
                workload_size=workload_size,
                entity_version=entity_version,
            )
        ]

        if not endpoint_exists:
            self.workspace.serving_endpoints.create(
                name=self.endpoint_name,
                config=EndpointCoreConfigInput(
                    served_entities=served_entities,
                ),
            )
        else:
      self.workspace.serving_endpoints.update_config(name=self.endpoint_name, served_entities=served_entities)

After the endpoint is deployed, we can view it in the workspace UI:

Making Predictions

After deployment, you can send prediction requests to your endpoint. The code demonstrates how to sample records from your test set, format them as JSON, and call the endpoint using Python’s requests library:

import requests

def call_endpoint(record):
    """
    Calls the model serving endpoint with a given input record.
    """
    serving_endpoint = f"{os.environ['DBR_HOST']}/serving-endpoints/marvel-character-model-serving/invocations"
    
    print(f"Calling endpoint: {serving_endpoint}")
    
    response = requests.post(
        serving_endpoint,
        headers={"Authorization": f"Bearer {os.environ['DBR_TOKEN']}"},
        json={"dataframe_records": record},
    )
    return response.status_code, response.text

sampled_records = test_set[required_columns].sample(n=18000, replace=True)
sampled_records = sampled_records.replace({np.nan: None}).to_dict(orient="records")

dataframe_records = [[record] for record in sampled_records]

status_code, response_text = call_endpoint(dataframe_records[0])

This enables you to validate your endpoint with real data and integrate it into downstream applications.

A/B Testing with Databricks Model Serving

Serving a single model is great, but how do you compare two models in production? A/B testing is a critical technique for continuous improvement and experimentation.

Databricks supports traffic split, by defining the list of served entities and percentage of traffic that goes to these entities. A common misconception is that simply splitting traffic between model versions qualifies as A/B testing but that’s not accurate. True A/B testing requires that a customer consistently sees the same model version throughout the experiment. This consistency is critical for accurately measuring performance differences (click-through rate, conversion, etc.).

Naive traffic split (like Databricks’ built-in routing) doesn’t guarantee this. A user might hit model A on one request and model B on the next. Sticky assignment is needed: Assign each user to a model variant based on a stable identifier (e.g., user ID or session ID), so they always get the same model during the test.

Let’s demonstrate how this can be implemented.

In the notebook notebooks/lecture6.ab_testing.py, we train two different models (A and B), each with its own configuration or hyperparameters:

from marvel_characters.models.basic_model import BasicModel

basic_model_a = BasicModel(config=config, tags=tags, spark=spark)
basic_model_a.train()

basic_model_a.log_model()

basic_model_a.register_model()

model_A_uri = f"models:/{basic_model_a.model_name}@latest-model"

basic_model_b = BasicModel(config=config, tags=tags, spark=spark)

basic_model_b.parameters = {"learning_rate": 0.01, "n_estimators": 1000, "max_depth": 6}

basic_model_b.model_name = f"{catalog_name}.{schema_name}.marvel_character_model_basic_B"

basic_model_b.train()

basic_model_b.log_model()

basic_model_b.register_model()

model_B_uri = f"models:/{basic_model_b.model_name}@latest-model"

Then, we define a custom MLflow PyFunc model (MarvelModelWrapper) to route requests to Model A or Model B based on a hash of a unique identifier (e.g., Id). This ensures a consistent split for reproducibility:

import hashlib

class MarvelModelWrapper(mlflow.pyfunc.PythonModel):

   def load_context(self, context):
      self.model_a = mlflow.sklearn.load_model(context.artifacts["sklearn-pipeline-model-A"])
      self.model_b = mlflow.sklearn.load_model(context.artifacts["sklearn-pipeline-model-B"])
      def predict(self, context, model_input):
      page_id = str(model_input["Id"].values[0])
      hashed_id = hashlib.md5(page_id.encode(encoding="UTF-8")).hexdigest()

      if int(hashed_id, 16) % 2:

          predictions = self.model_a.predict(model_input.drop(["Id"], axis=1))
          return {"Prediction": predictions[0], "model": "Model A"}

      else:
          predictions = self.model_b.predict(model_input.drop(["Id"], axis=1))
          return {"Prediction": predictions[0], "model": "Model B"}

The context must be passed at the model logging step (just as we did in lecture 4). Once the wrapper is registered and deployed in the same way we showed earlier, prediction requests are automatically split between the two models. You can monitor which model served each prediction and analyze results in the Databricks UI or programmatically.

Key Takeaways

In this lecture, we deployed our registered MLflow model to a Databricks Model Serving endpoint and exposed it as a REST API. We implemented a custom PyFunc wrapper to control the response payload and demonstrated how it can also be used for A/B testing by routing live traffic between different model versions.

Finally, we validated the endpoint in real time, showing how this approach supports experimentation while keeping deployment fast and reproducible.

With this setup, our model is not only live but also ready for iterative improvements in production.

This article is part of the course series, where we walk through the tools, patterns, and best practices for building and deploying machine learning workflows on Databricks.

This is lecture 5, where we talk about model serving architectures on Databricks. View it on Marvelous MLOps YouTube channel:

Model serving is a challenging topic for many machine learning teams. In an ideal scenario, the same team that develops a model, should be responsible for model deployment. However, this is not always feasible due to the knowledge gap or organizational structure. In that scenario, once model is ready, it is handed over to another team for deployment. It creates a lot of overhead when it comes to debugging and communication.

That’s where Databricks model serving can help a lot. Databricks model and feature serving use serverless, which simplifies the infrastructure side of the deployment, and a model endpoint can be created with one Python command (using Databricks sdk). It allows data science teams to own the deployment end-to-end and minimize the dependence on other teams.

In this article, we’ll discuss the following architectures:

• serving batch predictions (feature serving)

• model serving

• model serving with feature lookup

Feature serving

Serving batch predictions is probably one of the most popular and underestimated types of machine learning model deployment. Predictions are computed in advance using a batch process, stored in an SQL or in-memory database, and retrieved at request.

This architecture is very popular in the case of personal recommendation with low latency requirements. For example, an e-commerce store may recommend products to customers on various pages of the website.

Databricks Feature Serving is a perfect fit here. A scheduled Lakeflow job preprocesses data, retrains the model, and writes predictions to a feature table in Unity Catalog. These features are synced to an Online Store and exposed through a Feature Serving endpoint, defined by a FeatureSpec, a blueprint that combines feature functions (how features are calculated) with feature lookups (how they’re retrieved). Your application can then query this endpoint in real time to get fresh features for inference.

Model serving

Model serving assumes that model is deployed behind and endpoint, and all features are available through the payload. This is not the most realistic scenario, but can be still used in certain use cases, when models are embedded into apps and rely on user input from the app.

The figure below illustrates an automated model retraining and deployment pipeline on Databricks. A scheduled Lakeflow job preprocesses the data and retrains the model, which is then registered in Unity Catalog. The latest model is used to update a real-time serving endpoint, ensuring that applications always have access to the most up-to-date model.

This is the use case we cover in this course, and this is what we will deploy in the follow-up lecture, as Databricks Online store is not supported in Databricks Free edition.

Model serving with feature lookup

Model serving with feature lookup is the most complex (and most realistic) scenario that combines the ones we went through already. This architecture can be used for fraud detection, more complex recommendations, and many other use cases.

Here, some input features are passed with a request, some are retrieved from an Online Store (those features can be computed in a batch or streaming fashion, depending on your needs).

The figure below shows a simplified architecture for model serving with feature lookup. A scheduled Lakeflow job preprocesses data and updates a feature table in Unity Catalog, which is then synced to the online store. Another scheduled job retrains the model using the latest features and registers it in Unity Catalog. During inference, the model serving endpoint fetches features from the online store using a primary key, combines them with any features provided in the request payload, runs the model’s prediction function, and returns the result to the application.

Conclusion

Databricks now supports a wide range of machine learning model deployment architectures. Even though they come with some limitations (for example, you have limited influence on the payload structure), it significantly simplifies the deployment.

In the next lecture, we’ll dive into implementation of model serving endpoint and show how the registered model can be used to expose real‑time predictions for downstream applications.

This article is part of the course series, where we walk through the tools, patterns, and best practices for building and deploying machine learning workflows on Databricks.

Let’s dive into lecture 4 where we talk about logging and registering models with MLflow. View the lecture on Marvelous MLOps YouTube channel:

In the previous lecture, we have logged metrics, parameters, various artifacts, but have not logged a model yet. You could just saved a model in a .pkl file, but MLflow goes beyond that: it provides a standardized format called an MLflow Model, which defines how a model, its dependencies, and its code are stored. This is essential for downstream tasks like real-time serving, which will be covered later in the course.

A model can be logged using the mlflow.<model_flavor>.log_model() function. MLflow supports a wide range of flavors, such as lightgbm, prophet, pytorch, sklearn, xgboost, and many more. It also supports any custom model logics through PythonModel base class , which can be logged using pyfunc flavor.

Basic model: log, train, and register

To demonstrate logging, we’ll start with training a scikit-learn pipeline (referred to as Basic model) and logging it using sklearn flavor. We’ll walk through the notebooks/lecture4.train_register_basic_model.py code from the course GitHub repo.

Since we are interacting with MLflow, we need to set up tracking and registry URIs just as we did in lecture 3:

import mlflow
import os
from dotenv import load_dotenv

def is_databricks():
    return "DATABRICKS_RUNTIME_VERSION" in os.environ

if not is_databricks():
    load_dotenv()
    profile = os.environ["PROFILE"]
    mlflow.set_tracking_uri(f"databricks://{profile}")
    mlflow.set_registry_uri(f"databricks-uc://{profile}")

Then we’ll load the project configuration, initialize the SparkSession, and define tags we’ll need to tag the MLflow run and registered model:

from pyspark.sql import SparkSession

from marvel_characters.config import ProjectConfig, Tags

config = ProjectConfig.from_yaml(config_path="../project_config_marvel.yml", env="dev")
spark = SparkSession.builder.getOrCreate()
tags = Tags(**{"git_sha": "abcd12345", "branch": "main"})

We’ll need those to initialize an instance of BasicModel class. Then we load the data, prepare features, train and log the model:

from marvel_characters.models.basic_model import BasicModel

basic_model = BasicModel(config=config,
                         tags=tags,
                         spark=spark)

basic_model.load_data()
basic_model.prepare_features()

basic_model.train()

basic_model.log_model()

Let’s go through the logics behind the BasicModel class to understand what’s going on. After the class gets initialized, we set certain class attributes such as features, target, parameters, and model name.

We load the train and the test set using pyspark, and we’ll need these pyspark dataframes later to log the model input, together with the delta table version we retrieve. We also use toPandas() command to create pandas dataframes which are used for model training and evaluation.

Note that toPandas() command is rather inefficient, and if your dataset is large, you may want to look for alternatives, such as using deltatable package and external credentials vending in the way described in an earlier article. Logging input data in this case can be quite challenging.

import mlflow
import pandas as pd
from delta.tables import DeltaTable
from lightgbm import LGBMClassifier
from loguru import logger
from mlflow import MlflowClient
from mlflow.models import infer_signature
from pyspark.sql import SparkSession
from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline

from marvel_characters.config import ProjectConfig, Tags

class BasicModel:
    """A basic model class for Marvel character survival prediction 
       using LightGBM.
    """

    def __init__(self, config: ProjectConfig, 
                 tags: Tags, spark: SparkSession) -> None:

        self.config = config
        self.spark = spark
        self.tags = tags.to_dict()

        # Extract settings from the config
        self.num_features = self.config.num_features
        self.cat_features = self.config.cat_features
        self.target = self.config.target
        self.parameters = self.config.parameters
        self.catalog_name = self.config.catalog_name
        self.schema_name = self.config.schema_name
        self.experiment_name = self.config.experiment_name_basic
        self.model_name = f"{self.catalog_name}.{self.schema_name}.marvel_character_model_basic"
    
def load_data(self) -> None:
        """Load training and testing data from Delta tables.
        """
        logger.info("🔄 Loading data from Databricks tables...")
        self.train_set_spark = self.spark.table(f"{self.catalog_name}.{self.schema_name}.train_set")
        self.train_set = self.train_set_spark.toPandas()
        self.test_set_spark = self.spark.table(f"{self.catalog_name}.{self.schema_name}.test_set")
        self.test_set =  self.test_set_spark.toPandas()

        self.X_train = self.train_set[self.num_features + self.cat_features]
        self.y_train = self.train_set[self.target]
        self.X_test = self.test_set[self.num_features + self.cat_features]
        self.y_test = self.test_set[self.target]
        self.eval_data = self.test_set[self.num_features + self.cat_features + [self.target]]

        train_delta_table = DeltaTable.forName(self.spark,
                                               f"{self.catalog_name}.{self.schema_name}.train_set")
        self.train_data_version = str(train_delta_table.history().select("version").first()[0])
        test_delta_table = DeltaTable.forName(self.spark,
                                               f"{self.catalog_name}.{self.schema_name}.test_set")
        self.test_data_version = str(test_delta_table.history().select("version").first()[0])
        logger.info("✅ Data successfully loaded.")

The next method defined in the class is prepare_features(), which defines the sklearn pipeline that consists of 2 steps: encoding categorical variables using a custom encoder CatToIntTransofrmer, and LGBMClassifier.

LightGBM supports integer-encoded categorical features, which generally performs better than one-hot encoding. A custom encoder is necessary to make sure the LightGBM model treats integer-encoded features as categorical features, and earlier unseen categories get value -1 assigned to avoid errors while computing predictions.

You may notice that the CatToIntTransformer class is defined inside the prepare_features method. While this isn’t ideal from a design standpoint, it keeps the model self-contained, and we do not need to log our private package together with the model if we want to use the model for the downstream tasks. We’ll show a better to handle private dependencies when we discuss a custom pyfunc model later in this article.

def prepare_features(self) -> None:
        """Encode categorical features and define a preprocessing pipeline.
        """
        logger.info("🔄 Defining preprocessing pipeline...")

        class CatToIntTransformer(BaseEstimator, TransformerMixin):
            """Transformer that encodes categorical columns as 
               integer codes for LightGBM.

            Unknown categories at transform time are encoded as -1.
            """

            def __init__(self, cat_features: list[str]) -> None:
                """Initialize the transformer with categorical feature names."""
                self.cat_features = cat_features
                self.cat_maps_ = {}

            def fit(self, X: pd.DataFrame, y=None) -> None:
                """Fit the transformer to the DataFrame X."""
                self.fit_transform(X)
                return self

            def fit_transform(self, X: pd.DataFrame, y=None) -> pd.DataFrame:
                """Fit and transform the DataFrame X."""
                X = X.copy()
                for col in self.cat_features:
                    c = pd.Categorical(X[col])
                    # Build mapping: {category: code}
                    self.cat_maps_[col] = dict(zip(c.categories, 
                                      range(len(c.categories)), strict=False))
                    X[col] = X[col].map(lambda val, col=col: self.cat_maps_[col].get(val, -1)).astype("category")
                return X

            def transform(self, X: pd.DataFrame) -> pd.DataFrame:
                """Transform the DataFrame X by encoding categorical features as integers."""
                X = X.copy()
                for col in self.cat_features:
                    X[col] = X[col].map(lambda val, col=col: self.cat_maps_[col].get(val, -1)).astype("category")
                return X

        preprocessor = ColumnTransformer(
            transformers=[("cat", CatToIntTransformer(self.cat_features), self.cat_features)],
            remainder="passthrough"
        )
        self.pipeline = Pipeline(
            steps=[("preprocessor", preprocessor),
                   ("classifier", LGBMClassifier(**self.parameters))])
        logger.info("✅ Preprocessing pipeline defined.")

The train() fits the pipeline, and the log_model() method logs the model with all the required information:

• Signature is inferred using model input (X_train) and model output (the result of running the predict function on the pipeline), and passed when logging the model. If the signature is not provided, we would not be able to register model in Unity Catalog later.

• Input datasets (train and test sets, including the delta table version) are logged under the MLflow run to ensure that we can get the exact version of data used for training and evaluation, even if data was modified later, thanks to the time travel functionality of delta tables. Remember to set a proper retention period on the delta table (default is 7 days), otherwise you may not be able to access the exact version of the table if VACUUM command was executed. Most accounts have predictive optimization enabled by default, which means that Databricks automatically executes it as part of the optimization process.

 def train(self) -> None:
        """Train the model."""
        logger.info("🚀 Starting training...")
        self.pipeline.fit(self.X_train, self.y_train)

    def log_model(self) -> None:
        """Log the model using MLflow."""
        mlflow.set_experiment(self.experiment_name)
        with mlflow.start_run(tags=self.tags) as run:
            self.run_id = run.info.run_id

            signature = infer_signature(model_input=self.X_train,
                                        model_output=self.pipeline.predict(self.X_train))
            train_dataset = mlflow.data.from_spark(
                self.train_set_spark,
                table_name=f"{self.catalog_name}.{self.schema_name}.train_set",
                version=self.train_data_version,
            )
            mlflow.log_input(train_dataset, context="training")
            test_dataset = mlflow.data.from_spark(
                self.test_set_spark,
                table_name=f"{self.catalog_name}.{self.schema_name}.test_set",
                version=self.test_data_version,
            )
            mlflow.log_input(test_dataset, context="testing")
            self.model_info = mlflow.sklearn.log_model(
                sk_model=self.pipeline,
                artifact_path="lightgbm-pipeline-model",
                signature=signature,
                input_example=self.X_test[0:1]
            )

            result = mlflow.models.evaluate(
                    self.model_info.model_uri,
                    self.eval_data,
                    targets=self.config.target,
                    model_type="classifier",
                    evaluators=["default"],
                )
            self.metrics = result.metrics

Notice that we do not log any metrics. The metrics gets computed and logged under the same run using the mlflow.models.evaluate() function, which requires model URI, evaluation data, target, model type, and evaluators to run. Here, we use default evaluators, which means that standard metrics from the default evaluator gets logged:

After the model is logged, we can get the logged model using the model id (we can also use model id in the model URI to load the model):

logged_model = mlflow.get_logged_model(basic_model.model_info.model_id)
model = mlflow.sklearn.load_model(f"models:/{basic_model.model_info.model_id}")

This was not possible before MLflow 3, which introduced the concept of the LoggedModel. We also now have a separate model tab under the MLflow experiments in the UI. Let’s inspect the LoggedModel class. On purpose, I removed some metrics from the illustration (in fact, there is a separate entry for each metric shown in the table from the UI earlier).

It’s possible to access the model’s metrics and parameters (we have not logged any) directly from the LoggedModel class (which was only possible via the MLflow run in earlier versions of MLflow):

logged_model.params
logged_model.metrics

We still nee the run object to retrieve the information about the dataset inputs which were used to train and to evaluate the model:

run = mlflow.get_run(basic_model.run_id)


inputs = run.inputs.dataset_inputs
training_input = next((x for x in inputs if len(x.tags)>0 and x.tags[0].value == 'training'), None)
training_source = mlflow.data.get_source(training_input)
training_source.load()

testing_input = next((x for x in inputs if len(x.tags)>0 and x.tags[0].value == 'testing'), None)
testing_source = mlflow.data.get_source(testing_input)
testing_source.load()

The BasicModel class has another method, register_model(), which registers model in the Unity Catalog, together with the provided tags.

def register_model(self) -> None:
        """Register model in Unity Catalog."""
        logger.info("🔄 Registering the model in UC...")
        registered_model = mlflow.register_model(
            model_uri=self.model_info.model_uri,
            name=self.model_name,
            tags=self.tags,
        )
        logger.info(f"✅ Model registered as version {registered_model.version}.")

        latest_version = registered_model.version

        client = MlflowClient()
        client.set_registered_model_alias(
            name=self.model_name,
            alias="latest-model",
            version=latest_version,
        )
        return latest_version

Notice that we set the “latest-model” alias to make it easy to find the latest version of the registered model. “Latest” is a reserved value for the alias and can’t be used, and models can’t be referred as “latest” either.

Searching for model versions is pretty hard otherwise: you can only search by model name or alias. Searching using filter strings is not supported when model is registered in Unity Catalog.

Wrapping the model using pyfunc

Model signature in MLflow defines how different interfaces interact with the model. For instance, it defines the payload of the endpoint if the model gets served using Databricks model serving.

We’ve just registered a sklearn pipeline. If we deploy it behind an endpoint and query it, we will get an output in the format: {“Predictions”: [0]}. A pyfunc model flavor becomes useful if we want to adjust the model payload.

There are other scenarios when you may want to use a pyfunc. For example, if we need to access other systems (for example, a database) to return predictions, or if model serving requires specific artifacts (other files or even models).

Essentially, we are using pyfunc as a wrapper (In a certain sense, it’s very similar to the functionality of a FastAPI). Keeping the definition of the payload separate from the model itself is convenient : we can easily adjust the pyfunc wrapper definition without touching the registered model itself.

Let’s demonstrate how a pyfunc wrapper can be used. Under the custom_model module of the marvel-characters package, we defined the MarvelModelWrapper class. It has the load_context method which loads the basic model we trained earlier. The basic model gets loaded from the context, which gets stored together with the logged pyfunc model when we run the mlflow.pyfunc.log_model() function.

Notice that the predict method uses the adjust_predictions function defined outside of the MarvelModelWrapper, which means that the marvel_characters package must be now logged together with the pyfunc wrapper.

from datetime import datetime

import mlflow
import numpy as np
import pandas as pd
from mlflow import MlflowClient
from mlflow.models import infer_signature
from mlflow.pyfunc import PythonModelContext
from mlflow.utils.environment import _mlflow_conda_env

from marvel_characters.config import Tags


def adjust_predictions(predictions):
    return {"Survival prediction": ["alive" if pred == 1 else "dead" for pred in predictions]}

class MarvelModelWrapper(mlflow.pyfunc.PythonModel):

    def load_context(self, context: PythonModelContext) -> None:
        self.model = mlflow.sklearn.load_model(
            context.artifacts["lightgbm-pipeline"]
        )

    def predict(self, context: PythonModelContext, model_input: pd.DataFrame | np.ndarray) -> dict:
        predictions = self.model.predict(model_input)
        return adjust_predictions(predictions)

Let’s take a look at the log_register_model method. It takes the code_paths argument, which contains a local path to the marvel_characters package wheel. We use this list to define the conda_env. The location of the wheel in the artifacts folder (it will be saved in the code folder) is defined as a dependency.

Both code_paths and conda_env must be passed as an argument to the mlflow.pyfunc.log_model() function. Here, we also pass artifacts, which is a dictionary that contains the basic model URI.

def log_register_model(self, wrapped_model_uri: str, pyfunc_model_name: str,
                           experiment_name: str, tags: Tags, code_paths: list[str],
                           input_example: pd.DataFrame) -> None:
        mlflow.set_experiment(experiment_name=experiment_name)
        with mlflow.start_run(run_name=f"wrapper-lightgbm-{datetime.now().strftime('%Y-%m-%d')}",
            tags=tags.to_dict()):
            additional_pip_deps = []
            for package in code_paths:
                whl_name = package.split("/")[-1]
                additional_pip_deps.append(f"code/{whl_name}")
            conda_env = _mlflow_conda_env(additional_pip_deps=additional_pip_deps)

            signature = infer_signature(model_input=input_example,
                                        model_output={"Survival prediction": ["alive"]})
            model_info = mlflow.pyfunc.log_model(
                python_model=self,
                name="pyfunc-wrapper",
                artifacts={
                    "lightgbm-pipeline": wrapped_model_uri},
                signature=signature,
                code_paths=code_paths,
                conda_env=conda_env,
            )
        client = MlflowClient()
        registered_model = mlflow.register_model(
                model_uri=model_info.model_uri,
                name=pyfunc_model_name,
                tags=tags.to_dict(),
            )
        latest_version = registered_model.version
        client.set_registered_model_alias(
            name=pyfunc_model_name,
            alias="latest-model",
            version=latest_version,
        )
        return latest_version

The pyfunc wrapper gets registered in the same way as the basic model, here we also set the “latest-model” alias. This is how we log and register the pyfunc wrapper in notebooks/lecture4.train_register_custom_model.py:

from importlib.metadata import version

marvel_characters_v = version("marvel_characters")

code_paths=[f"../dist/marvel_characters-{marvel_characters_v}-py3-none-any.whl"]

client = MlflowClient()
wrapped_model_version = client.get_model_version_by_alias(
    name=f"{config.catalog_name}.{config.schema_name}.marvel_character_model_basic",
    alias="latest-model")

test_set = spark.table(f"{config.catalog_name}.{config.schema_name}.test_set").toPandas()
X_test = test_set[config.num_features + config.cat_features]

pyfunc_model_name = f"{config.catalog_name}.{config.schema_name}.marvel_character_model_custom"
wrapper = MarvelModelWrapper()
wrapper.log_register_model(wrapped_model_uri=f"models:/{wrapped_model_version.model_id}",
                           pyfunc_model_name=pyfunc_model_name,
                           experiment_name=config.experiment_name_custom,
                           input_example=X_test[0:1],
                           tags=tags,
                           code_paths=code_paths)

After the pyfunc model is logged and registered, we can see in the UI how its artifacts are stored. We can find the basic model’s artifacts in the artifacts folder, and the package wheel in the code folder. Notice that the wheel is referenced in the requirements.txt. When the environment gets created, all the dependencies of our private package get installed.

The model can be loaded using the mlflow.pyfunc.load_model() function. If we want to access the original MarvelModelWrapper class and its attributes, we must use the unwrap_python_model() method.

We can run the predict function after we loaded the model. However, this does not guarantee that the model will be loaded successfully at the serving step. That’s because we are utilizing our existing environment.

loaded_pufunc_model = mlflow.pyfunc.load_model(f"models:/{pyfunc_model_name}@latest-model")

unwraped_model = loaded_pufunc_model.unwrap_python_model()

unwraped_model.predict(context=None, model_input=X_test[0:1])

There is a more reliable way that mimics the creation of model serving environment. Note that this code only runs from Databricks environment and would not work in the VS Code.

predictions = mlflow.models.predict(
    f"models:/{pyfunc_model_name}@latest-model",
    X_test[0:1])

Conclusions

In this lecture, we went beyond logging metrics and parameters, and logged and registered a model with MLflow. We made sure to capture the model signature, dataset versions, and tags containing the code version (for now, just a dummy value) so our runs are fully reproducible.

We registered the model in Unity Catalog and wrapped it in a pyfunc to control the output and package extra dependencies for serving.

Next up, we’ll dive into model serving architectures and see how all of this comes together in production.

This article is part of the course series, where we walk through the tools, patterns, and best practices for building and deploying machine learning workflows on Databricks.

Let’s dive into lecture 3, where we talk about MLflow experiment tracking. View the lecture on Marvelous MLOps YouTube channel:

MLflow is probably the most popular tool for model registry and experiment tracking out there. MLFlow is open source and integrates with a lot of platforms and tools.

Due to its extensive support and a lot of options, getting started with MLflow may feel overwhelming. In this lecture, we will get back to the basics, and will review 2 most important classes in MLFlow that form the foundation of everything else, mlflow.entities.Experiment and mlflow.entities.Run.

We will see how those entities get created, how you can retrieve them, and how they change based on different input parameters. In this course, the Databricks version of MLflow is used, so it contains some Databricks-specific information. However, the idea is generalizable to any MLflow instance.

Before we go any further, let’s discuss how we can authenticate towards MLflow tracking server on Databricks.

Tracking URI

By default, MLflow will track experiment runs using the local file systems, and all the metadata will be stored in the ./mlruns directory. We can verify that by retrieving the current tracking URI:

import mlflow
mlflow.get_tracking_uri()

In lecture 2, we explained how we can authenticate towards Databricks using Databricks CLI, which we continued to use when developing in VS Code. Now we must make MLflow aware of it, and use Databricks MLflow tracking server.

This can be done by calling mlflow.set_tracking_uri().Even though we’re only using experiment tracking for now, starting with MLflow 3, it’s also necessary to set the registry URI using mlflow.set_registry_uri(). There are a couple of things to pay attention to:

• The tracking and the registry URI must contain the profile that you used to log in (which is defined in the .databrickscfg file, for exampledbc-1234a567-b8c9).

• We want multiple developers with a different profile name to collaborate on the same code base.

• We only want to set the tracking and the registry URI when running code outside of Databricks.

This leads us to the following possible solution. We can then define whether the code runs within a Databricks environment by checking that the DATABRICKS_RUNTIME_VERSION environment variable is available. Every developer can store the profile name in .env file (which is ignored by git via .gitignore file), and set environment variable PROFILE using Python package dotenv:

import os
import mlflow
from dotenv import load_dotenv

def is_databricks() -> bool:
    """Check if the code is running in a Databricks environment."""
    return "DATABRICKS_RUNTIME_VERSION" in os.environ

if not is_databricks():
    load_dotenv()
    profile = os.environ.get("PROFILE")
    mlflow.set_tracking_uri(f"databricks://{profile}")
    mlflow.set_registry_uri(f"databricks-uc://{profile}")

This is how the content of the .env file would look like:

# content of the .env file
PROFILE=dbc-1234a567-b8c9

Now we use Databricks MLflow as tracking server, we can create an experiment.

MLflow Experiment

Experiments in MLflow are the main unit of organization for ML model training runs. All MLflow runs belong to an experiment.

An MLflow experiment can be created using either the mlflow.create_experiment or mlflow.set_experimentcommand. The latter is more commonly used, as it will activate an existing experiment by name or create a new one if it doesn't already exist.

It is possible to add tags to your experiment by passing them as a parameter to the mlflow.create_experiment command. If the experiment already exists, tags can be added using the mlflow.set_experiment_tags command after the experiment has been activated:

experiment = mlflow.set_experiment(experiment_name="/Shared/marvel-demo")
mlflow.set_experiment_tags(
    {"repository_name": "marvelousmlops/marvel-characters"})

The above commands will result in the creation of an instance of mlflow.entities.Experiment class with the following attributes:

The MLflow experiment created above can be now found by name or by tag:

# search experiment by name
mlflow.search_experiments(filter_string="name='/Shared/demo'")

# search experiment by tag
mlflow.search_experiments(
    filter_string="tags.repository_name='marvelousmlops/marvel-characters'")

It is also possible to search for experiments by last_update_time or creation time. We can also get an experiment by experiment id:

mlflow.get_experiment(experiment.experiment_id)

MLflow Run

After the experiment is created, we can create an experiment run. An MLflow run is related to a single execution of ML model training code, under an MLflow run a lot of things can be logged: params, metrics, and artifacts of different formats.

An MLflow run is created using the mlflow.start_run command. This command is typically used within a with block to ensure proper handling of the run lifecycle. If you don’t use a with block, you must explicitly end the run by calling mlflow.end_run. Here, we log some metrics and a parameter:

with mlflow.start_run(
    run_name="marvel-demo-run",
    tags={"git_sha": "1234567890abcd"},
    description="marvel character prediction demo run",
) as run:
    run_id = run.info.run_id
    mlflow.log_params({"type": "marvel_demo"})
    mlflow.log_metrics({"metric1": 1.0, "metric2": 2.0})

The code above will result in the creation of an instance of mlflow.entities.Run class. It’s easy to visualize how the MLflow run class is constructed by saving the run information as a JSON file and using JSON Lens:

We can notice the git_sha tag that we passed to the run, and some tags were created by MLflow. The tags would be different depending on where this run was created. For example, if the run was created when running in a Lakeflow job, the job id and the task run id will be available as tags.

The run can be restarted by passing the run id to the mlflow.start_run() command. We can log extra information as part of the run, but we can’t overwrite existing metrics and parameters.

Logging and loading artifacts

Let’s now start a new run and log various artifacts:

import matplotlib.pyplot as plt
import numpy as np

mlflow.start_run(run_name="marvel-demo-run-extra",
                 tags={"git_sha": "1234567890abcd"},
                       description="marvel demo run with extra artifacts",)
mlflow.log_metric(key="metric3", value=3.0)
# dynamically log metric (trainings epochs)
for i in range(0,3):
    mlflow.log_metric(key="metric1", value=3.0+i/2, step=i)
mlflow.log_artifact("../demo_artifacts/mlflow_meme.jpeg")
mlflow.log_text("hello, MLflow!", "hello.txt")
mlflow.log_dict({"k": "v"}, "dict_example.json")
mlflow.log_artifacts("../demo_artifacts", artifact_path="demo_artifacts")

# log figure
fig, ax = plt.subplots()
ax.plot([0, 1], [2, 3])

mlflow.log_figure(fig, "figure.png")

# log image dynamically
for i in range(0,3):
    image = np.random.randint(0, 256, size=(100, 100, 3), 
                              dtype=np.uint8)
    mlflow.log_image(image, key="demo_image", step=i)

mlflow.end_run()

We can view the experiment run in the UI and find all the artifacts under the Artifacts tab:

We can also search for the run programmatically. Notice that we can filter our search results based on tags, run name, metrics, status, and tags:

from time import time

time_hour_ago = int(time() - 3600) * 1000

runs = mlflow.search_runs(
    search_all_experiments=True, #or experiment_ids=[], or experiment_names=[]
    order_by=["start_time DESC"],
    filter_string="status='FINISHED' AND "
                  f"start_time>{time_hour_ago} AND "
                  "run_name LIKE '%marvel-demo-run%' AND "
                  "metrics.metric3>0 AND "
                  "tags.mlflow.source.type!='JOB'"

We can now load the artifacts using the appropriate MLflow load functions, and download all artifacts using the download_artifacts() function.

artifact_uri = runs.artifact_uri[0]
mlflow.artifacts.load_dict(f"{artifact_uri}/dict_example.json")

mlflow.artifacts.load_image(f"{artifact_uri}/figure.png")

mlflow.artifacts.download_artifacts(
    artifact_uri=f"{artifact_uri}/demo_artifacts",
    dst_path="../downloaded_artifacts")

To simplify your understanding of logging and loading artifacts in MLflow, we prepared a cheatsheet. In the next lecture, we’ll go through logging and registering models.

If you are interested to go broader and dive deeper into the topic, get your code reviewed by us, and get constant feedback, follow our End-to-end MLOps with Databricks course on Maven.

Use code MARVELOUS for 100 euro off.

Temporary discount (until 10 August 2025): BDAY for 20% off.

This article is part of the course series, where we walk through the tools, patterns, and best practices for building and deploying machine learning workflows on Databricks.

Let’s dive into lecture 2, where we talk about developing on Databricks. View the lecture on Marvelous MLOps YouTube channel:

Most people using Databricks start by developing directly in a Databricks notebook, because it’s easy, fast, and convenient. But when it comes to MLOps, that convenience can quickly become a bottleneck. Notebooks make it difficult to write modular code, apply proper code quality standards, or run unit tests, all of which are essential for maintainable, production-grade ML systems.

Fortunately, there’s a better way. Databricks developer tools, such as VS Code extension, Databricks CLI, and Databricks Connect, allow you to develop locally using modern engineering workflows, while still running your pyspark code on Databricks.

In this lecture, we’ll show you how to use these tools to move development outside notebooks and adopt workflows that align better with MLOps practices.

Getting started

To follow along with the course, you’ll need to set up a few things. In the video, we demonstrate how to walk through these steps.

1.Get the Databricks free edition. The course uses Databricks free edition. Do not confuse it with Databricks free trial, it is not the same thing!

2.Fork the course repo: https://github.com/marvelousmlops/marvel-characters. Forking the repo will allow you to work on CI/CD pipeline later in the course. Clone the forked repo on your local machine

3.Create catalogs and schemas. In the Databricks free edition workspace, create catalogs mlops_dev, mlops_acc, and mlops_prd.Under each catalog, create schema marvel_characters. These catalogs and schemas are required to run the code.

4.Install the CLI. Databricks has very good documentation on how to install it: https://docs.databricks.com/en/dev-tools/cli/install.html. From our experience, the homebrew option works great on MacOS. On Window — winget. Otherwise, you can always install from a source build.

5.Authenticate towards Databricks. Databricks CLI should be used to authenticate towards Databricks from your local machine.

We do not recommend using personal access tokens (from a security perspective, this is not the best option). Instead, use user-to-machine authentication. For the workspace-level commands, use the following command to authenticate:

databricks auth login --host <workspace-url>

It creates an authentication profile in .databrickscfg file that looks like this (this is not a real host, just an example):

[dbc-1234a567-b8c9]
host      = https://dbc-1234a567-b8c9.cloud.databricks.com/
auth_type = databricks-cli

6.Install Databricks VS Code extension: follow the official documentation. Once installed, update the databricks.yml file to match the host of the Databricks Free edition workspace.

Local Python environment management

In this course, we use uv for Python project management, and we highly recommend trying it out if you haven’t already.

The Python version and project dependencies are defined in the pyproject.toml file. You’ll notice that we specify exact versions for all packages. This is intentional, but only because the project is not meant to be reused as a library. For reusable packages, locking dependencies too strictly can lead to dependency resolution issues down the line.

In our case, the marvel-characters package is meant to be self-contained. When we install it inside a Databricks environment, we want these specific dependency versions to override the existing ones.

[project]
name = "marvel-characters"
description = "Marvel characters project"
requires-python = ">=3.12, <3.13"
dependencies = [
    "mlflow==3.1.1",
    "cffi==1.17.1",
    "cloudpickle==3.1.1",
    "numpy==1.26.4",
    "pandas==2.3.0",
    "pyarrow==17.0.0",
    "scikit-learn==1.7.0",
    "lightgbm==4.6.0",
    "scipy==1.16.0",
    "databricks-sdk==0.55.0",
    "pydantic==2.11.7",
    "loguru==0.7.3",
    "python-dotenv==1.1.1",
]

dynamic = ['version']

[project.optional-dependencies]
dev = ["databricks-connect>=16.3, <17",
       "ipykernel>=6.29.5, <7",
       "pip>=25.0.1, <26",
       "pre-commit>=4.1.0, <5"]

ci = ["pre-commit>=4.1.0, <5"]

For development purpose, we use dev optional dependency. To get started, you just need to run:

uv sync --extra dev

This command will build Python virtual environment, stored in .venv inside your project folder.

If you want to execute pyspark code on Databricks, you need to choose the cluster in the VS Code extension UI. In the Free edition that we have set up, you can only choose serverless compute. When working locally, our local Python version environment defines how the code runs. If you want to run the code in a Databricks notebook, you need to choose the corresponding serverless environment.

Serverless environments

Currently, Databricks supports three environment versions, each comes with its own requirements. you can view the release notes for more details.

What matters is that your project’s Python version must match the environment Python version. In our case, we use Python 3.12, which matches environment version 3, which comes with the following system settings:

• Operating System: Ubuntu 24.04.2 LTS

• Python: 3.12.3

• Databricks Connect: 16.3.2

It also has certain versions of Python packages installed. Note that we are not using those package versions. For example, we use MLflow 3.1.1 instead of 2.21.3.

To see how to choose the environment version in a Databricks notebook and install the project’s dependencies, you can use Git Folder functionality. Databricks documentation walks you through the process of creating the Git folder.

After the Git folder is created, go to the notebooks folder of the project. Note that all the files inside the notebooks folder are .py files, with the first line “# Databricks notebook source”, and cells delimited by “# COMMAND — — — — — ”.

Open the lecture2.marvel_data_preprocessing.py notebook. To select an environment version: click the Environment side panel in the notebook UI, select version 3, and click apply.

To install the project dependencies, run:

%pip install -e ..
%restart_python

The project code will not be importable unless you adjust the path (which should be avoided in any other scenario).

from pathlib import Path
import sys
sys.path.append(str(Path.cwd().parent / 'src'))

Instead of using Git Folder, you can also syncronize your project folder to a location in the workspace using the sync button under “Remote Folder” in the Databricks VS Code extension.

This allows to install package in a nicer way: you can build the package locally using uv build. If the resulting wheel file is not ignored by .gitignore, it will be synced together with other files, allowing to install the wheel directly within the notebook.

Now you can run the project code in a Databricks notebook using the serverless environment 3 and our project’s dependencies. We can also execute it directly in the VS code. Let’s take a look at the code we are executing in a detail.

Project code

In this course, we are using the Marvel Characters dataset from Kaggle. Later in the course, we are going to use the character’s features to predict whether the character is going to survive. All the features, along with the model parameters, experiment names, and schema/ catalogs used in the projects, are stored in the project_config_marvel.yml file:

prd:
  catalog_name: mlops_prd
  schema_name: marvel_characters
acc:
  catalog_name: mlops_acc
  schema_name: marvel_characters
dev:
  catalog_name: mlops_dev
  schema_name: marvel_characters


experiment_name_basic: /Shared/marvel-characters-basic
experiment_name_custom: /Shared/marvel-characters-custom

parameters:
  learning_rate: 0.01
  n_estimators: 1000
  max_depth: 6


num_features:
  - Height
  - Weight
  
cat_features:
  - Universe
  - Identity
  - Gender
  - Marital_Status
  - Teams
  - Origin
  - Magic
  - Mutant

target: Alive

On purpose, we keep project configurations in a separate file, to make adjustments easily when needed.

Throughout the prokject, we use a ProjectConfig class to manage and validate project-level configuration. The configuration is validated using the pydantic library, which ensures correct data types and structure at runtime.

from typing import Any

import yaml
from pydantic import BaseModel


class ProjectConfig(BaseModel):
    """Represent project configuration parameters loaded from YAML.

    Handles feature specifications, catalog details, and experiment parameters.
    Supports environment-specific configuration overrides.
    """

    num_features: list[str]
    cat_features: list[str]
    target: str
    catalog_name: str
    schema_name: str
    parameters: dict[str, Any]
    experiment_name_basic: str | None
    experiment_name_custom: str | None

    @classmethod
    def from_yaml(cls, config_path: str, env: str = "dev") -> "ProjectConfig":
        """Load and parse configuration settings from a YAML file.

        :param config_path: Path to the YAML configuration file
        :param env: Environment name to load environment-specific settings
        :return: ProjectConfig instance initialized with parsed configuration
        """
        if env not in ["prd", "acc", "dev"]:
            raise ValueError(f"Invalid environment: {env}. Expected 'prd', 'acc', or 'dev'")

        with open(config_path) as f:
            config_dict = yaml.safe_load(f)
            config_dict["catalog_name"] = config_dict[env]["catalog_name"]
            config_dict["schema_name"] = config_dict[env]["schema_name"]

            return cls(**config_dict)

The lecture2.marvel_data_preprocessing.py notebook also starts with loading the configuration:

import pandas as pd
import yaml
from loguru import logger
from pyspark.sql import SparkSession

from marvel_characters.config import ProjectConfig
from marvel_characters.data_processor import DataProcessor

config = ProjectConfig.from_yaml(config_path="../project_config_marvel.yml", env="dev")

logger.info("Configuration loaded:")
logger.info(yaml.dump(config, default_flow_style=False))

After loading the configuration, we initialize a SparkSession. Note that SparkSession is imported from the pyspark library, but we don’t have pyspark as a project dependency. That’s because we're using databricks-connect, which includes pyspark internally. This is why these two libraries cannot be installed side by side as they would conflict with each other.

Next, we load the dataset from the data/ folder and create an instance of the DataProcessor class. This class is responsible for preprocessing the data, splitting it into training and test sets, and writing the processed data to Unity Catalog. This ensures traceability and makes it easy to reference the datasets consistently across environments.

Finally, we enable change data feed on the resulting Delta tables, which allows you to track row-level changes between versions:

spark = SparkSession.builder.getOrCreate()

df = pd.read_csv("../data/marvel_characters_dataset.csv")

data_processor = DataProcessor(df, config, spark)
data_processor.preprocess()

X_train, X_test = data_processor.split_data()
data_processor.save_to_catalog(X_train, X_test)
data_processor.enable_change_data_feed()

The preprocess() method performs data cleaning and normalization to prepare the Marvel character dataset for training. It includes the following steps:

• Column renaming for consistency

• Filling missing values in categorical columns like Universe, Origin, Identity, and Gender

• Grouping rare Universe categories under “Other”

• Converting the Teams column to a binary indicator based on whether the value is present

• Creating new binary features such as Magic and Mutant by detecting keywords in the Origin column

• Normalizing values in the Origin column into a fixed set of categories like Human, Mutant, Asgardian, Alien, etc.

• Converting the Alive column into a binary target variable and filtering only valid values

• Keeping only the selected numeric and categorical features, along with the target and PageID (which is renamed to Id)

This preprocessing step produces a clean, well-structured dataset that’s ready for model training. We’ll use it in Lecture 4, where we focus on training, logging, and registering a model. But before that, in the next lecture, we’ll explore MLflow experiment tracking, the foundation of everything that happens within MLflow.

If you are interested to go broader and dive deeper into the topic, get your code reviewed by us, and get constant feedback, follow our End-to-end MLOps with Databricks course on Maven.

Use code MARVELOUS for 100 euro off.

Temporary discount (until 10 August 2025): BDAY for 20% off.

This article is part of the course series, where we walk through the tools, patterns, and best practices for building and deploying machine learning workflows on Databricks :

• Lecture 1: Introduction to MLOPs

• Lecture 2: Developing on Databricks

• Lecture 3: Getting started with MLflow

• Lecture 4: Log and register model with MLflow

• Lecture 5: Model serving architectures

• Lecture 6: Deploying model serving endpoint

• Lecture 7: Databricks Asset Bundles

• Lecture 8: CI/CD and deployment strategies

• Lecture 9: Intro to monitoring

• Lecture 10: Lakehouse monitoring

Let’s dive into lecture 1. View the lecture recording on Marvelous MLOps YouTube:

If you’ve worked with machine learning in a real-world setting, you’ve likely heard the term MLOps. According to Wikipedia:

“MLOps is a paradigm that aims to deploy and maintain machine learning models in production reliably and efficiently.”

But what does production actually mean in this context?

In practice, production means that the output of a machine learning model is consistently delivered to end users or systems ,and that it drives real business value.

Here’s a simple example:

A data scientist is asked to build a demand forecasting model. They develop a proof of concept in a Databricks notebook, more than a thousand lines of code, that trains a model, generates predictions, and writes those predictions to a Delta table.

That Delta table is then used by the fulfillment team to order products. Since forecasts need to be updated weekly, the data scientist schedules the notebook to run once a week.

Is this in production?
Yes. The model’s outputs are actively used to support business decisions.
Is it efficient?
To some extent. Automating the process with a scheduled notebook is certainly more efficient than running everything manually.
Is it reliable?
Not really.

This setup lacks testing, monitoring, error handling, version control, and formal deployment processes. If anything breaks, the data, the code, or the environment, there may be no alert, no rollback, and no audit trail. To address these gaps, we turn to MLOps principles, a set of practices designed to bring reliability, transparency, and control to machine learning workflows.

MLOps principles

Done right, MLOps enables teams to move faster without sacrificing stability or accountability.

Traceability and reproducibility is the main principle in MLOps. For any ML model deployment or run, we should be able to look up unumbibuously:

• Corresponding code/ commit on git.

• Infrastructure/ environment used for training and serving.

• Data used for model training.

• Generated ML model artifacts.

This is just one set of MLOps principles. In practice, there are several other core pillars that contribute to building robust, maintainable, and collaborative machine learning systems:

• Documentation: Every ML project should include clear documentation outlining business goals, KPIs, architectural decisions, and model selection rationale. Good documentation ensures continuity, helping onboard new team members, supporting handovers when someone leaves, and enabling other teams to understand how to integrate with the system.

• Code Quality: Maintaining high code quality is essential for stability and collaboration. This includes enforcing code reviews on all pull requests, adhering to style and structure conventions, and incorporating automated testing (unit, integration, and ideally ML-specific tests). A clean, testable codebase makes experimentation safer and deployments more reliable.

• Monitoring: In ML projects, monitoring extends beyond system metrics to include model performance. Offline evaluation metrics should be logged and tracked over time. In production, business KPIs and model-related costs must also be monitored. This provides visibility into how the model behaves in the real world and helps detect unintended outcomes early.

These principles, when applied consistently, form the foundation of a healthy ML project, which not only delivers value now, but remains stable and adaptable as teams, data, and requirements evolve.

MLOps tooling

Following MLOps principles isn’t feasible without the right tooling. While it might initially seem like you need dozens of different tools, the truth is: you need categories of tools, not necessarily specific ones. And chances are, your organization already has many of them in place.

Here’s a breakdown of the essential categories:

• Version Control: Tools like GitHub, GitLab, Bitbucket are the foundation of reproducibility and collaboration in any ML project.

• CI/CD: Jenkins, GitHub Actions, GitLab CI/CD, and Azure DevOps help automate testing, packaging, and deployment of ML code and artifacts.

• Orchestration: Apache Airflow, Argo Workflows, or Lakeflow jobs coordinate complex multi-step ML workflows.

• Model Registry: Platforms like MLflow or Comet track model versions and metadata.

• Feature Stores or data version control tools: tools like Feast, Hopsworks, or Databricks’ built-in feature store ensure consistent feature definitions across training and inference.

• Compute & Serving: whether you’re running training jobs on Databricks, SageMaker, or Kubernetes, or serving models via REST endpoints, your infrastructure must be able to support both development and production needs.

• Monitoring: most machine learning projects require custom metrics to monitor, along with the standard application performance metrics, which allows to use common observability tools like Prometheus and Grafana or ELK stack.

• Container Registry: tools like DockerHub, Azure Container Registry, or AWS ECR are used to store and manage Docker images for consistent packaging and deployment of ML applications

In MLOps, each tool is there to support MLOps principles. Focus must be on covering essential capabilities, not collecting tools.

Here’s what each tool category enables:

• Version control, orchestration, CI/CD: Code reproducibility and traceability, and code quality.

• Model registry: Model traceability and reproducibility.

• Feature store, data versioning: Data traceability and reproducibility.

• Container registry: Environment traceability and reproducibility

A simple, purpose-driven stack is all you need to do MLOps well.

Is not it all just DevOps?

At first glance, the tools and principles of MLOps might seem identical to DevOps, and in many ways, they are. MLOps builds on DevOps but adapts it to the unique challenges of machine learning.

DevOps aims to break the wall of confusion between development and IT operations to enable faster, more reliable delivery. MLOps shares that goal, but the wall is thicker and taller. On one side, you have ML teams focused on experimentation and modeling; on the other, platform teams focused on infrastructure and reliability. These groups often speak different technical languages. ML teams lack operations experience, and DevOps teams may not understand the specifics of ML systems.

This disconnect makes the development and deployment cycle for ML systems even more challenging, and that’s exactly the gap MLOps is designed to close.

But there’s a deeper difference. In traditional software systems, consistent code, infrastructure, and environment yield consistent results. In ML systems, a change in the statistical properties of the data alone can lead to different behavior. That’s why data monitoring and retraining aren’t just operational extras, they’re core to doing MLOps right.

MLOps on Databricks

MLOps isn’t about tools, it’s about principles. But you do need the right tools to follow those principles effectively. A few years ago, I might have recommended building your own platform. But today, the landscape has changed.

We’re in the middle of a tooling explosion in data and AI. There are more specialized solutions than ever before, but managing them can quickly become overwhelming. Databricks now brings together all the essential components for MLOps in one platform. While each individual part may not be the absolute best on the market, the value lies in having everything tightly integrated and ready to use.

Here’s how Databricks supports key MLOps functions:

• Lakeflow Jobs for orchestration

• MLflow for experiment tracking and Unity Catalog for model registry

• Delta Tables for data versioning, also powering feature tables

• Databricks Compute for model training

• Feature Serving and Model Serving for real-time inference

• Lakehouse Monitoring for detecting data and model drift

• Databricks Asset Bundles to streamline CI/CD and deployment

With these components in place, you can focus on building reliable ML systems, without stitching together a dozen tools yourself.

In this course, we’ll deep dive into all these features and explain how they can be used together to promote traceability, reproducibility, and monitoring standards.

If you are interested to go broader and dive deeper into the topic, get your code reviewed by us, and get constant feedback, follow our End-to-end MLOps with Databricks course on Maven.

Use code MARVELOUS for 100 euro off.

Temporary discount (until 10 August 2025): BDAY for 20% off.

He runs a course on the LLM software development lifecycle (I am joining the July cohort!), focusing on everything from retrieval and evaluation to agent design, and all the intermediate steps in between.

Enough talking, I’ll let him dig into today’s controversial topic: “Stop building AI agents”. ↓🎙️

Hugo: I’ve taught and advised dozens of teams building LLM-powered systems. There’s a common pattern I keep seeing, and honestly, it’s frustrating.

Everyone reaches for agents first. They set up memory systems. They add routing logic. They create tool definitions and character backstories. It feels powerful, and it feels like progress.

Until everything breaks. And when things go wrong (which they always do), nobody can figure out why.

Was it the agent forgetting its task? Is the wrong tool getting selected? Too many moving parts to debug? Is the whole system fundamentally brittle?

I learned this the hard way. Six months ago, I built a “research crew” with CrewAI: three agents, five tools, perfect coordination on paper. But in practice? The researcher ignored the web scraper, the summarizer forgot to use the citation tool, and the coordinator gave up entirely when processing longer documents. It was a beautiful plan falling apart in spectacular ways.

This flowchart came from one of my lessons after debugging countless broken agent systems. Notice that tiny box at the end? That’s how rarely you actually need agents. Yet everyone starts there.

This post is about what I learned from those failures, including how to avoid them entirely.

The patterns I’ll walk through are inspired by Anthropic’s Building Effective Agents post. It’s not just theory. This is real code, real failures, and real decisions I’ve made while teaching these systems. Every example here comes from actual projects I’ve built or debugged.

You’ll discover why agents aren’t the answer (most of the time). And more importantly, you’ll learn what to build instead.

What You’ll Learn:

• Why agents are usually not the right first step

• Five LLM workflow patterns that solve most problems

• When agents are the right tool and how to build them safely

🔗 All examples come from this GitHub notebook

Don’t Start with Agents

Everyone thinks agents are where you start. It’s not their fault: frameworks make it seem easy, demo videos are exciting, and tech Twitter loves the hype.

But here’s what I learned after building that CrewAI research crew: most agent systems break down from too much complexity, not too little.

In my demo, I had three agents working together:

• A researcher agent that could browse web pages

• A summarizer agent with access to citation tools

• A coordinator agent that managed task delegation

Pretty standard stuff, right? Except in practice:

• The researcher ignored the web scraper 70% of the time

• The summarizer completely forgot to use citations when processing long documents

• The coordinator threw up its hands when tasks weren’t clearly defined

So wait: “What exactly is an agent?” To answer that, we need to look at 4 characteristics of LLM systems.

When people say “agent,” they mean that last step: the LLM output controls the workflow. Most people skip straight to letting the LLM control the workflow without realizing that simpler patterns often work better. Using an agent means handing control to the LLM. But unless your task is so dynamic that its flow can’t be defined upfront, that kind of freedom usually hurts more than it helps. Most of the time, simpler workflows with humans in charge still outperform full-blown agents.

I’ve debugged this exact pattern with dozens of teams:

1. We have multiple tasks that need automation

2. Agents seem like the obvious solution

3. We build complex systems with roles and memory

4. Everything breaks because coordination is harder than we thought

5. We realize simpler patterns would have worked better

🔎 Takeaway: Start with simpler workflows like chaining or routing unless you know you need memory, delegation, and planning.

Workflow patterns you should use

These five patterns come from Anthropic’s taxonomy — implemented, tested, and demoed in my notebook:

Use case: “Writing personalized outreach emails based on LinkedIn profiles.”

You want to reach out to people at companies you’re interested in. Start by extracting structured data from a LinkedIn profile (name, role, company), then generate a tailored outreach email to start a conversation.

Here are 3 simple steps:

1. Turn raw LinkedIn profile text into structured data (e.g., name, title, company):

linkedin_data = extract_structured_data(raw_profile)

2. Add relevant company context for personalization (e.g., mission, open roles):

company_context = enrich_with_context(linkedin_data)

3. Generate a personalized outreach email using the structured profile + company context:

email = generate_outreach_email(linkedin_data, company_context)

Guidelines:

✅ Use when: Tasks flow sequentially
⚠️ Failure mode: Chain breaks if one step fails
💡 Simple to debug, predictable flow

(2) Parallelization

Use case: Extracting structured data from profiles

Now that chaining works, you want to process many profiles at once and speed up the processing. Split each profile into parts — like education, work history, and skills, then run extract_structured_data() in parallel.

Here are 2 simple steps:

1. Define tasks to extract key profile fields in parallel:

tasks = [ extract_work_history(profile), # Pull out work experience details extract_skills(profile), # Identify listed skills extract_education(profile) # Parse education background ]

2. Run all tasks concurrently and gather results:

results = await asyncio.gather(*tasks)

Guidelines:

✅ Use when: Independent tasks run faster concurrently
⚠️ Failure mode: Race conditions, timeout issues
💡 Great for data extraction across multiple sources

(3) Routing

Use case: LLM classifies the input and sends it to a specialized workflow

Say you’re building a support tool that handles product questions, billing issues, and refund requests. Routing logic classifies each message and sends it to the right workflow. If it’s unclear, fall back to a generic handler.

Here are 2 simple steps:

1. Choose a handler based on profile type:

if profile_type == "executive": handler = executive_handler() # Use specialized logic for executives elif profile_type == "recruiter": handler = recruiter_handler() # Use recruiter-specific processing else: handler = default_handler() # Fallback for unknown or generic profiles

2. Process the profile with the selected handler:

result = handler.process(profile)

Guidelines:

✅ Use when: Different inputs need different handling
⚠️ Failure mode: Edge cases fall through routes
💡 Add catch-all routes for unknowns

(4) Orchestrator-Worker

Use case: LLM breaks down the task into 1 or more dynamic steps

You’re generating outbound emails. The orchestrator classifies the target company as tech or non-tech, then delegates to a specialized worker that crafts the message for that context.

Here are 2 simple steps:

1. Use LLM to classify the profile as tech or non-tech:

industry = llm_classify(profile_text)

2. Route to the appropriate worker based on classification:

if industry == "tech": email = tech_worker(profile_text, email_routes) else: email = non_tech_worker(profile_text, email_routes)

The orchestrator-worker pattern separates decision-making from execution:

At first glance, this might resemble routing: a classifier picks a path, then a handler runs. But in routing, control is handed off entirely. In this example, the orchestrator retains control: it initiates the classification, selects the worker, and manages the flow from start to finish.

This is a minimal version of the orchestrator-worker pattern:

You can scale this up with multiple workers, sequential steps, or aggregation logic (and I encourage you to! If you do so, make a PR to the repository), but the core structure stays the same.

Guidelines:

✅ Use when: Tasks need specialized handling
⚠️ Failure mode: Orchestrator delegates subtasks poorly or breaks down the task incorrectly
💡 Keep orchestrator logic simple and explicit

(5) Evaluator-Optimizer

Use case: Refining outreach emails to better match your criteria

You’ve got an email generator running, but want to improve tone, structure, or alignment. Add an evaluator that scores each message and, If it doesn’t pass, send it back to the generator with feedback and loop until it meets your bar.

Here are 2 simple steps:

1. Generate an initial email from the profile:

content = generate_email(profile)

2. Loop until the email passes the evaluator or hits a retry limit:

while True: score = evaluate_email(content) if score.overall > 0.8 or score.iterations > 3: break content = optimize_email(content, score.feedback)

Guidelines:

✅ Use when: Output quality matters more than speed
⚠️ Failure mode: Infinite optimization loops
💡 Set clear stop conditions

🔎 Takeaway: Most use cases don’t need agents. They need better workflow structure.

When to Use Agents (If You Really Have To)

Agents shine when you have a sharp human in the loop. Here’s my hot take: agents excel at unstable workflows where human oversight can catch and correct mistakes.

When agents actually work well:

Example 1: Data Science Assistant

An agent that writes SQL queries, generates visualizations, and suggests analyses. You’re there to evaluate results and fix logical errors. The agent’s creativity in exploring data beats rigid workflows.

To build something like this, you’d give the LLM access to tools like run_sql_query(), plot_data(), and summarize_insights(). The agent routes between them based on the user’s request — for example, writing a query, running it, visualizing the result, and generating a narrative summary. Then, it feeds the result of each tool call back into another LLM request with its memory context. We walk through a live example of this pattern in our Building with LLMs course.

Example 2: Creative Writing Partner

An agent brainstorming headlines, editing copy, and suggesting structures. The human judges quality and redirects when needed. Agents excel at ideation with human judgment.

Example 3: Code Refactoring Assistant

Proposing design patterns, catching edge cases, and suggesting optimizations. The developer reviews and approves changes. Agents spot patterns humans miss.

When NOT to use agents

Enterprise Automation
Building stable, reliable software? Don’t use agents. You can’t have an LLM deciding critical workflows in production. Use orchestrator patterns instead.

Back to my CrewAI research crew: the agents kept forgetting goals and skipping tools. Here’s what I learned:

Failure Point #1: Agents assumed they had context that they didn’t
Problem: Long documents caused the summarizer to forget citations entirely
What I’d do now: Use explicit memory systems, not just role prompts

Failure Point #2: Agents failed to select the right tools
Problem: The researcher ignored the web scraper in favor of a general search
What I’d do now: Constrain choices with explicit tool menus

Failure Point #3: Agents did not handle coordination well
Problem: The coordinator gave up when tasks weren’t clearly scoped
What I’d do now: Build explicit handoff protocols, not free-form delegation

🔎 Takeaway: If you’re building agents, treat them like full software systems. Don’t skip observability.

• ❌ Agents are overhyped and overused

• 🔁 Most cases need simple patterns, not agents

• 🤝 Agents excel in human-in-the-loop scenarios

• ⚠️ Don’t use agents for stable enterprise systems

• 🧪 Build with observability and explicit control

Agents are overhyped and often overused. In most real-world applications, simple patterns and direct API calls work better than complex agent frameworks. Agents do have a role-in particular, they shine in human-in-the-loop scenarios where oversight and flexibility are needed. But for stable enterprise systems, they introduce unnecessary complexity and risk. Instead, aim to build with strong observability, clear evaluation loops, and explicit control.

Want to go deeper? I teach a course on the entire LLM software development lifecycle (use code MARVELOUS10 for 10% off), covering everything from retrieval and evaluation to observability, agents, and production workflows. It’s designed for engineers and teams who want to move fast without getting stuck in proof of concept purgatory.

If you’d like a taste of the full course, I’ve put together a free 10-part email series on building LLM-powered apps. It walks through practical strategies for escaping proof-of-concept purgatory: one clear, focused email at a time.

Copyrights: The article was originally published in collaboration with Jordan Cutler on High Growth Engineer — original article.

If not otherwise stated, all images are created by the author.

Bubbles in technology are not new. In 2007, an a capella group The Richter Scales created a song "Here comes another bubble". They received an Webby Award for Viral Video. Today, this video is more relevant than ever!

I decided to create a parody on this song. Lyrics is mostly written by me (ChatGPT is not great in understanding the beat). The song is made with AI!

Lyrics:

Almost got a CS degree
Stanford dropout, dreaming free
Watched the hype and took the bait
Now I’m coding AI fate

November thirtieth, it dropped
ChatGPT — the world has stopped
Everyone just paused and stared
Google watched — but wasn't scared

GPT-4 made its mark
Claude appeared, polite and smart
Google scrambled, shipped out Bard
(Launching LLMs is hard)

Here comes another bubble
AI’s gone full throttle
Scraping every novel

Couldn’t sleep, was up all night
Turned my scripts into a site
Chased the trend — it never ends
Now I'm running CodeWithFriends

Left my crypto days behind
Bragged I’ve built a thinking mind
AI agent, it's so hot
With MCP — I kid you not

Dreamed I'd close a massive seed
"AGI" was all I'd need
Even though it’s held with tape
VCs came to seal the fate

Here comes another bubble
The VCs are buzzing
AI does everything (but nothing)

Launched a demo late one night
Woke up, X has lost its mind
Angels circled, term sheets flew
“Pre-revenue? We love that too”

Did demo live, it didn’t crash
Woke up trending, raised more cash
Forbes 30 Under 30 call
Didn’t know I’d peaked at all

Blog it, film it, make those reels
TikTok trends and viral feels
Tweet it, thread it, demo day
Build a bot that writes your way

Here comes another bubble
In a year, we’ll pivot
AGI — we live it

“Helps you code” but wrecks your build
Kills the flow, your patience spilled
System broken, things unclear
Keep believing, A-G-I is near

Billion tokens burned in vain
Models hallucinate again
Now your startup’s pivoted
To crème brûlée with AI flame

Here comes another bubble
And when we are gone
This will still go on
And on and on and on
And on and on
POP!

This can be particularly useful for machine learning projects, which utilize libraries like scikit-learn that expect Pandas or Polars dataframe as an input. If the source data is stored as a delta table in Unity Catalog, a standard approach of first creating a pyspark dataframe and then transforming it into a Polars or Pandas dataframe can be very inefficient. Instead, we would like to read data from Unity Catalog without going through pyspark.

In this article, we show how to achieve that. We also challenge the idea of using pyspark or spark SQL for data transformation (which is common for data preprocessing pipelines), showing that using Polars can help to achieve serious speedup and cost savings for certain data sizes.

Benchmark & requirements

For the benchmark we used the industry standard TPC-H benchmark: https://www.tpc.org/tpch/. The functions are the ones as used again from the repository from Polars: https://github.com/pola-rs/polars-benchmark.

All the code for the article can be found in the repository. For your convenience, we made all the data available via a shared folder and prepared the scripts to dump the files in a Volume, and create the external tables with the feature of deleteVectors turned off. You can run the notebooks yourself to experience the difference in performance with minimal effort.

These are the steps:

• Clone the repository into your Databricks workspace using the Git folder feature. It is a public repository, so you do not have to authenticate

• Update project_config file to use your preferred catalog and schema

• Toggle ‘External Data Access’ via the Metastore Toggle in the Catalog Explorer (Figure 1).

• Grant the EXTERNAL_USE_SCHEMA privilege to your user. This will be exposed once the toggle above has switched. Note: ALL_PRIVILEGES does not include the EXTERNAL_USE_SCHEMA privilege (Figure 2).

• Run notebooks/init_benchmark_uc.py script (you need to have permissions to create schemas and volumes) to download the data from the shared folder and create delta tables

• Execute notebooks/run_notebooks.py using Serveless cluster vs classic compute.

To read data from Unity Catalog, we will use the mechanism of credential vending. It works using short-living tokens and entirely bypasses the dependency of Spark. In the notebook, we use a personal access token that we retrieve using dbutils. For production, you need to create a service principal and store its token in a secret scope.

Findings

As expected, on smaller datasets (x1, x3 scale) Polars runs faster than PySpark (Figure 3). Up to 2.5 times as fast (as measured by rerunning the TPC-H benchmark as defined in the repo for the Polars benchmark).

The problem is, it doesn’t scale well (serverless supports max 32 gb RAM). With the demonstrated approach you load all the data directly into the local RAM and execute. There are ways to compute out-of-core, but at the time of writing not possible with reliability: the short-living token connection with Unity Catalog does not seem very reliable. For some reason the load data command keeps hanging without timing out, you really need to reset the entire instance for it to work again. When running the code locally, this issue does not seem to be present.

When it comes to the costs, on x1 scale on serverless (32 gb), using polars is at least 15x cheaper than running the same workload using Spark (total cost is 0,183 dollar vs 3,267 dollar, for 15 runs). Figure 4 shows the distribution of individual runs. On a larger scale (x10), using Spark SQL would be more cost effective.

Downsides

1. Reading from Unity Catalog. Currently (20-05-2025) Polars doesn’t seem to support it to read directly from Unity Catalog: an obscure error is raised that mentions the URL, despite trying various methods. However, what did work is to read the table via the library deltalake. This can be used to expose an arrow dataset, which subsequently can be loaded into Polars. Probably in the near future direct read can be supported.

2. Writing to Unity Catalog. This seems not yet possible as both the Deltalake and Polars libraries raise an error when trying. However, it can be done by writing directly to the storage as delta table, and creating an external table referring to that data.

3. Breaking the lineage. Databricks' Unity Catalog captures post-facto lineage at runtime. You need to have executed code, for it to be able to pick up what the lineage between data objects was. However, this only works for pyspark. Databricks is working on the Bring Your Own Lineage feature, which will allow users to update the lineage themselves. This would solve, with effort, the downside of running an external engine: losing the lineage. For machine learning projects, we could tag MLflow experiment runs, passing information about the table location, and the version of delta table.

Conclusions

Reading data directly from a delta table without using Spark is a great idea if you do not need Spark for your workload (for example, for a machine learning workflow). It is possible to run the code anywhere: on your local PC, standard cluster, or serverless.

For data preprocessing, it may be beneficial to use polars too (for smaller datasets). You can save quite some costs – at a cost. The cost being that you lose the sweet features that the Spark on Databricks provides; mostly the lineage, but also that not all features of the tables are supported and need to be turned off. What you save can be quite impressive, though: running a (small) load on Serverless, you can in the smallest instance reach a speed up of 3x on Serverless in comparison with PySpark, or a 10x speedup when running locally on a Macbook M4 Pro.

We believe that the machine learning teams should be responsible for machine learning model deployment end-to-end. However, they often lack the skills, especially when it comes to the real-time model serving. Luckily, there are tools that aim to simplify that part of the deployment, if you are a Databricks user. Databricks model serving is worth considering for that purpose: models can be deployed with minimal code using Python SDK, given that the model training is tracked using MLflow and registered in Unity Catalog.

It all sounds simple, but has a downside: if something is wrong with your deployment (especially if you are using a custom model), you will only see it after you have waited for 15-20 minutes. Very time-consuming (and expensive) interaction cycle… But it does not have to be that way! Databricks model serving utilizes MLflow model serving, which means, you can also test it locally.

In this article, we’ll demonstrate the local testing workflow using the well-known Iris-Species dataset, focusing on three critical steps:

1. Download a pyfunc model artifact from MLflow experiment tracking.

2. Deploy the model endpoint locally.

3. Test the endpoint.

Before we proceed with these steps, let’s walk through the prerequisites and code we used to train the Iris model and log it in MLflow.

Prerequisites

First of all, make sure you have Databricks CLI installed. Follow the instructions if it is not the case. Authenticate to your Databricks workspace. As a result, you will have a .databrickscfg file with a profile.

The code for this article can be found in this repository: https://github.com/marvelousmlops/iris-mlflow-serve-testing.

Clone the repository, and set up your virtual environment with all the dependencies by running the following command:

uv sync –-all-extras

Model training & MLflow experiment tracking

First of all, we get the data and train the sklearn pipeline to predict the Iris species.

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split


iris = datasets.load_iris(as_frame=True)
X = iris.data
y = iris.target
logger.info("The dataset is loaded.")


X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=28)
preprocessor = ColumnTransformer(
               transformers=[("std_scaler", StandardScaler(), iris.feature_names)])


pipeline = Pipeline(steps=[("preprocessor", preprocessor), 
                          ("classifier", LogisticRegression()),
                          ])
pipeline.fit(X_train, y_train)

After that, we define a custom model wrapper:

import pandas as pd
import numpy as np
from typing import Union
from mlflow.pyfunc import PythonModelContext


class ModelWrapper(mlflow.pyfunc.PythonModel):
   def __init__(self, model: object) -> None:
       self.model = model
       self.class_names = iris.target_names


   def predict(
       self,
       context: PythonModelContext,
       model_input: Union[pd.DataFrame, np.array], 
   ) -> Union[pd.DataFrame, np.ndarray]: 
       raw_predictions = self.model.predict(model_input)
       mapped_predictions = [self.class_names[int(pred)] for pred in raw_predictions]
       return mapped_predictions

This model wrapper is pretty basic: the only thing we change is that the pipeline outputs the name of the iris species instead of a number. Of course, you can put any logic in such a wrapper class and make it more complex.

Finally, we log the model together with some metrics and parameters. It is important to specify the profile in the MLflow tracking URI (it is the same as the profile name in the .databrickscfg file).

import mlflow
from loguru import logger
from mlflow.models import infer_signature

mlflow.autolog(disable=True)
# mlflow.set_tracking_uri(“databricks://your-profile”) - use it to run locally
mlflow.set_tracking_uri("databricks")
mlflow.set_experiment("/Shared/iris-demo")


with mlflow.start_run() as run:
   run_id = run.info.run_id
   y_proba = pipeline.predict_proba(X_test)


   # Evaluation metrics
   auc_test = roc_auc_score(y_test, y_proba, multi_class='ovr')
   logger.info(f"AUC Report: {auc_test}")


   # Log parameters and metrics
   mlflow.log_param("model_type", "LogisticRegression Classifier with preprocessing")
   mlflow.log_metric("auc", auc_test)


   # Log the model
   signature = infer_signature(model_input=X_train, model_output=["setosa"])
   dataset = mlflow.data.from_pandas(iris.frame, name="train_set")
   mlflow.log_input(dataset, context="training")


   mlflow.pyfunc.log_model(
       python_model=ModelWrapper(pipeline),
       artifact_path=f"pyfunc-lg-pipeline-model",
       signature=signature,
   )

Downloading the artifacts

This is how the logged model looks like in the Databricks workspace:

To download an artifact from your Databricks workspace to your local machine, you first need to configure your environment for secure access.

Start by setting the MLflow tracking URI to point to your Databricks workspace:

export MLFLOW_TRACKING_URI=databricks://<your profile name>

Then, authenticate using Databricks CLI:

databricks auth login --profile <your profile name>

Once authenticated, you can use the MLflow CLI to download artifacts. The following command downloads a specific artifact from a given MLflow run to a designated local directory:

mlflow artifacts download -r <mlflow_run_id> -a <artifact_name> -d <destination>

For example, to download a model artifact named `pyfunc-lg-pipeline-model`from a particular run, you might use:

uv run mlflow artifacts download -r b4bad079af4649dba6fffc51f33b8902 -a pyfunc-lg-pipeline-model -d .

This flow ensures that ML artifacts tracked in Databricks are easily retrievable for local testing, validation, or deployment pipelines, supporting robust ML operations and reliable API contract testing. This generates a local directory with the following structure inside iris-mlflow-serve-testing project:

The `MLmodel` file defines the model’s flavors (e.g., `python_function` or `sklearn`), while `conda.yaml` and `requirements.txt` ensure environment reproducibility.

Local Model Deployment

At this stage, you have already set up a virtual environment as described in the prerequisites step. Make sure that pyenv is installed before you proceed.

Now you can serve the model through MLflow's local inference server on a custom port (5088 in this example):

uv run mlflow models serve -m pyfunc-lg-pipeline-model --port 5088

Now we are ready to test the model locally. The MLflow model server will listen for incoming requests on the specified port (5088). You can send requests to this endpoint to test the model's behavior.

Endpoint testing

At this early stage, testing the Inference API Contract — checking input/output formats, response consistency — helps ensure your model will work smoothly with downstream systems. Doing this early means you catch integration problems before scaling up to full deployment on Databricks, saving time and reducing headaches later.

MLflow Inference Server provides 4 endpoints: ping, health, version and invocations. As Databricks model serving differs from the local deployment, and we can only test to a certain extent, we are only interested in the /invocations endpoint that returns model’s predictions for a given input.

The /invocations endpoint accepts data in the dataframe_split of dataframe_records format (it is also possible to provide Tensor input, but that would require a different model specification, hence is not relevant for our example).

As mentioned in the documentation, it is not advised to use dataframe_records, let’s proceed with testing using dataframe_split instead.

Assume we have the following test_data and we need to convert them to a pandas dataframe so that we can pass the data to the endpoint.

BASE_URL = “http://127.0.1.1:5088”

test_data = {
    "sepal length (cm)": 6.1,
    "sepal width (cm)": 2.8,
    "petal length (cm)": 4.7,
    "petal width (cm)": 1.2,
}

pandas_df = pd.DataFrame([test_data])

payload_dataframe_split = json.dumps(
    {"dataframe_split": pandas_df.to_dict(orient="split")}
)

Let’s do the following tests:

• When the correct input (a single observation or a dataframe with multiple observations) is provided, we get status 200

• When we provide input with too few fields, we get status 400

Here is an example of one such test:

def test_inference_server_invocations() -> None:
   """Test model invocations using split dataframe format.
   Verifies successful response and valid prediction format."""

   response = requests.post(
       f"{BASE_URL}/invocations",
       data=payload_dataframe_split,
       headers={"Content-Type": "application/json"},
       timeout=2,
   )
   assert response.status_code == 200
   logger.info(f"Received {response.json()}")
   value = response.json()["predictions"]
   assert isinstance(value, list)

We can simply run `uv run pytest` and make sure all tests pass.

Conclusions

Deploying machine learning models can be tricky, especially when different teams handle model creation and deployment. It's easy to run into issues that take time to fix once the model is live.

Testing your models locally before deploying them to Databricks can save you time and headaches. By downloading your model, running it on your computer, and testing its inputs and outputs, you can catch problems early. This way, you can be more confident that your model will work correctly when deployed, leading to a smoother process overall. Tools like MLflow make this local testing easier. This approach means fewer surprises and a more reliable deployment.

To learn more about MLOps with Databricks, join our End-to-end MLOps with Databricks course.

It covers everything you need for an actual production-grade AI application, from developing on Databricks, experiment tracking with MLflow, registering models in Unity Catalog, deploying real-time endpoints, workflow orchestration, and CI/CD pipelines.

💰 Use code MARVELOUS to get 100 EUR off.

Humanity has always been drawn to utopia — a perfect, idealized future where all problems are solved. Believing that the world is steadily marching toward this vision is tempting. In the AI landscape, the unicorn (you have noticed the 5th leg, right?) represents the elevated promises, wild imagination, and relentless hype that paint a picture of transformative, almost magical technology.

The rainbow, however, represents the real world: entire potential but riddled with imperfections, inconsistencies, and systemic barriers.

Just like the stock market, AI has its declines and flows. Everything might seem to skyrocket, but a slight shift — technical debt, regulatory burdens, or enterprise realities — can send it crashing back to earth. The question is not whether AI is a transformative force (there is no doubt it is!) but whether we’re being realistic about its trajectory.

This article will discuss the reality of using AI in the enterprise environment, address technical debt, bridge knowledge gaps, and understand the herd effect that fuels the AI bubble. We aim to offer a realistic roadmap for businesses navigating the complex AI landscape by critically analyzing these factors.

1. The AI bubble

We have been in Data & AI for over 10 years. The AI bubble has never been so big. We have AI everywhere on our laptops, phones, and websites. The CEOs of Nvidia, Microsoft, Meta, and OpenAI are spreading a lot of news about revolutionary AI technology, how AI agents will replace humans, how we will reach AGI soon, and how we will have AI everywhere. We live in an AI bubble, and even though the technology is accurate, it is nontrivial to apply it to actual use cases and drive business value than advertised.

The technological advancements in the AI field are significant, and the value AI generates is real. However, there are still many gaps that people who try to build AI products see clearly. Two factors contribute to the AI bubble: knowledge and the herd effect. Somehow, they are tangential but different.

2. The Role of Knowledge Gaps

The gap between AI insiders and the general public is one of the hinds of the AI bubble. The saying “Knowledge is power” remains valid for AI within its development and implementation context.

People who are deeply invested in the development of AI are fully aware of the nuances, challenges, and limitations that come with the implementation of AI-based solutions.

On the other hand, AI outsiders are constantly awe-struck by the marketing terms associated with AI which presents an entirely different world of possibilities. This knowledge gap enables misconceptions to spread at an alarming rate, therefore making the hype of AI take precedence over the reality of what AI systems can offer.

3. Herd Effect: Fear of Missing Out (FOMO)

Another significant factor driving the AI hype is the herd effect or the Fear of Missing Out (FOMO).

As more companies invest in AI and tout their successes, others feel compelled to follow suit, fearing they’ll fall behind if they don’t adopt AI technologies. This rush often leads to deploying AI solutions without a thorough understanding of their applicability or potential ROI, further inflating the AI bubble. The result is a market saturated with AI buzzwords and solutions that may not deliver the promised transformative impact.

AI models (under AI models, we understand foundation models) are used everywhere, where a standard ML model should be used instead. This adds complexity and decreases reliability.

4. Back to basics

Most companies can not reliably bring standard machine learning models to production and lack monitoring practices.

Despite what many people think, workflows that include AI models are, on average, more complex to bring into production and monitor — even if you take the simplest scenario without RAG or finetuning involved — just call a 3rd party API.

In too many cases, we seem to have forgotten the basic principles of machine learning and blindly rely on what that API outputs. This is the danger of AI hype: AI has become accessible to everyone, and many software engineers treat it as just another API call.

What could go wrong? The data model has not changed, the code has not changed (and neither has the environment where it gets executed), and the version of the API has not changed. But this is the beauty of machine learning: even if everything in your control has not changed, the model can start performing poorly unexpectedly because data distribution has changed.

This does not just happen with standard machine learning models, it also happens with AI models — we just have less means to impact that behavior, and prompt finetuning becomes an essential part of the process.

5. Real-world, 2025

Experts say that 2025 will be the year of AI agents. But is it really true?

While the AI hype machine continues to boom, real-world adoption tells a different story. The promise of autonomous AI agents seamlessly operating across enterprises remains largely aspirational. The reality? AI in enterprise is still a work in progress — complex, expensive, and often misaligned with actual business needs.

Take BBVA, the Spanish bank that went all in on OpenAI’s technology. They deployed over 2,900 AI models to enhance productivity, yet integrating them into their existing systems turned out to be a logistical nightmare. AI doesn’t operate in a vacuum; it needs to connect with legacy infrastructure, existing workflows, and strict regulatory requirements. And that’s where reality bites — scaling AI across an enterprise is exponentially harder than rolling out a chatbot.

The UK government’s attempt to integrate AI into its welfare system faced significant limitations. At least six AI prototypes, designed to enhance staff training, improve job center services, and streamline disability benefit processing, were discontinued due to issues in scalability, reliability, and insufficient testing. Officials acknowledged several “frustrations and false starts,” highlighting the complexities involved in deploying AI within public services.

A study highlighted several obstacles in developing and deploying AI agents within enterprises. Security concerns were identified as a top challenge by leadership (53%) and practitioners (62%). Other significant challenges included data governance, performance issues, and integration complexity. These findings underscore the multifaceted difficulties organizations face in implementing AI agents effectively.

Reflecting on these examples, it’s evident that the widespread adoption of AI agents in enterprise settings faces significant limitations. While 2025 may usher in extensive research, proofs of concept (POCs), and minimum viable products (MVPs), the path to full-scale integration remains complex.

6. AI in a corporate environment

Big companies operate under strict rules, structured workflows, and a constant focus on ROI. Unlike agile startups that can adapt on the fly, large organizations have to deal with complex approval processes, compliance checks, and risk management. All this makes adopting AI a slower process, and the idea of rapid transformation often feels more like a distant dream than something achievable.

Chip Huyen references the most common LLM applications in her AI engineering book. Enterprises are risk-averse and prefer to deploy internal-facing applications first. From what we have seen so far, even though there is initial support from the leadership to deploy such applications, not enough funding goes to those projects (and unlikely will) as they do not generate direct business value. We are not saying there is no value — there is, but it is challenging to convince stakeholders.

In enterprises, the most common use cases with direct business generation are related to customer service (forwarding customers to the right agents/ processes) and reviewing contracts. These use cases have been there for a while, and have historically been NLP-heavy, and AI models helped to improve these projects.

Some companies have tried to use LLMs for recommendations and chatbots, and the world has seen enough failures. Here are some examples:

DPD’s customer-facing chatbot, “Ruby,” was designed to assist customers with their inquiries. However, due to insufficient safeguards, a user was able to provoke the bot into swearing and composing a poem criticizing the company itself. This incident underscores the importance of implementing strict content moderation protocols and regularly updating AI systems to prevent such occurrences.

Similarly, Pak’nSave’s AI meal planner app, intended to provide innovative recipe suggestions, malfunctioned and recommended a combination of ingredients that would produce chlorine gas, labeling it as an “aromatic water mix.” This highlights the critical need for rigorous testing and validation of AI outputs, especially in applications directly impacting consumer health and safety.

It feels like not everyone has learned from it, and we regularly see companies launching AI applications without clear business value with poor guardrails, mainly for “marketing purposes”. Let’s hope it will not turn out to be bad marketing, as users will try to make the app do things it is not supposed to do “just for fun”.

There are exceptions. Some companies created nice LLM-powered recommendations, for example, Zalando. It has well-implemented guardrails and is useful for the customers (it helps to find items that are otherwise hard to find via search). In October 2024, Zalando expanded its AI-powered assistant to all 25 markets, supporting local languages. This expansion aims to provide customers with personalized fashion advice and insights into emerging local trends, thereby enhancing the shopping experience.

7. Areas of attention & conclusions

There is great potential to leverage AI in a corporate setting. However, to hope for enterprise adoption, we must consider security gaps, controlled environments, transparency and traceability, and a way to monitor and evaluate AI systems.

In enterprise ecosystems, AI systems need large volumes of data, including personal and proprietary information. Their role is to enhance workflows and boost efficiency, but they need access to critical systems, which can be considered a security risk. Organizations must focus on preventing unapproved access to data, breaches, and compliance issues.

Threat actors can deploy malware that mimics AI behavior to breach networks, skew decisions, or steal secrets. AI agents act autonomously, making them harder to detect and control. This creates a major challenge: real-time oversight of AI systems.

Monitoring is a persistent issue. Few companies have proper systems in place, and AI’s complexity makes it even harder. Owners must fully understand every decision their AI makes

AI’s transformative potential is undeniable, but the path from hype to reality is complex and challenging. Rather than chasing unicorns and rainbows, organizations must take a grounded, strategic approach — one that prioritizes real business needs, robust security frameworks, and a deep understanding of AI’s limitations. The road ahead is uncertain, but one thing is clear: the way we answer these questions will determine whether AI becomes a lasting force for good or just another passing bubble.

What do you think — are we ready?

Developing on Databricks outside of Databricks environment is challenging, and there are 4 main developer tools Databricks provides:

• Databricks CLI: a command line interface that allows you to interact with Databricks platform. It is a very powerful tool with a large range of commands (essentially, all the functionality available via API or Terraform is also available via CLI).

• Databricks asset bundles: developers tools that allow you to simplify deployment of various assets on Databricks. Databricks bundle commands are part of the CLI. Check out a related article.

• Databricks Connect — a Python package (there is also support for Scala and R) that allows you to trigger an execution of spark code on a Databricks cluster from a local environment

• Databricks VS Code Extension — an extension that integrates with all 3 other tools: Databricks connect, Databricks asset bundles, and Databricks connect. Makes it very easy to connect to the Databricks workspace and execute your code using a Databricks cluster.

In this article, we focus on Databricks CLI, Databricks Connect, and VS Code Extension. We will not go through all the features of all these tools but will share some (not very obvious) findings that hopefully will help you in your development and debugging process.

Databricks CLI

1.Installing the CLI. Databricks has very good documentation on how to install it: https://docs.databricks.com/en/dev-tools/cli/install.html

From our experience, the homebrew option works great on MacOS. On Window — winget. Otherwise, you can always install from a source build.

2. Authentication. Databricks CLI should be used to authenticate towards Databricks from your local machine.

We do not recommend using personal access tokens (from a security perspective, this is not the best option). Instead, use user-to-machine authentication.

For the workspace-level commands, use the following command to authenticate:

databricks auth login --host <workspace-url>

It creates an authentication profile in .databrickscfg file that looks like this (this is not a real host, just an example):

[dbc-1234a567-b8c9] host      = https://dbc-1234a567-b8c9.cloud.databricks.com/ auth_type = databricks-cli

One thing to pay attention to is the evaluation order:

1. For any command run from the bundle working directory — the values of fields within a project’s bundle setting files.

2. The values of environment variables (DATABRICKS_HOST and DATABRICKS_TOKEN)

3. Configuration profile field values within the .databrickscfg file.

If you happen to have an environment variable set up, it might mess up with authentication.

Databricks Connect

Let’s try out Databricks Connect now without using VS Code Extension.

1.Create & start a cluster. We use the smallest Personal compute cluster with 15.4 LTS runtime:

2. Create a demo folder. Create a virtual environment and install databricks-connect package. We must use Python 3.11.x and databricks-connect version 15.4.x — to match the runtime.

mkdir demo cd demo  uv venv -p 3.11 .venv  source .venv/bin/activate  uv pip install databricks-connect==15.4.4  python

3. Run the code. Now we got into the Python console from the terminal. Let’s try to run some commands:

from databricks.connect import DatabricksSession  spark = DatabricksSession.builder.profile("dbc-1234a567-b8c9").getOrCreate()  df = spark.read.table("samples.nyctaxi.trips")  df.show(5)

A couple of things to note here:

1. You need to choose the correct profile from the .databrickscfg file. If the profile is named DEFAULT, it is not needed.

2. DatabricksSession will be used to run spark code. If you, for example, run df.toPandas() — which may be needed for an ML library you are using, the dataframe will be loaded into your machine’s memory. And if the data is too big, you will get out of memory errors.

3. It may surprise you, but you can also import pyspark — even though pyspark is not listed as a dependency of databricks-connect, and uv does not show that pyspark is installed.

It is because pyspark is “hidden” and embedded into databricks-connect (and that’s why it is a bad idea to list both databricks-connect and pyspark in your project dependencies)

You can see it by going to .venv/lib/python3.11/site-packages:

4. Let’s try to use SparkSession instead!

from pyspark.sql import SparkSession  spark = SparkSession.builder.getOrCreate()  df = spark.read.table("samples.nyctaxi.trips")  df.show(5)

It should not surprise you, that this would not work. Import works without issues (because pyspark is installed), but when we try to run the second line, we get the RuntimeError: “Only remote Spark sessions using Databricks Connect are supported. Use DatabricksSession.builder to create a remote Spark session instead.”

Interestingly enough, this works with VS Code Extension! Let’s get into it!

VS Code Extension

Let’s try out Databricks Connect now without using VS Code Extension.

1.Install the extension. Follow the official documentation.

2.Open the demo project. It does not have any files except .venv folder. Click on the Databricks logo -> create configuration -> choose your profile.

3. All green checkmarks: Activate your environment, install pip, and select your cluster.

4. Let’s check the files. We can see that The extension created several files, including databricks.yml file. The host in the file must match the host that is specified in the profile.

5. Let’s create a demo.py file (the same code we tried to run from a terminal). We’ll also add the first line “# Databricks notebook source”- and the code can be recognized as a notebook.

This works because the extension created configuration in the .vscode folder (settings.json):

{     "jupyter.interactiveWindow.cellMarker.codeRegex": "^# COMMAND ----------|^# Databricks notebook source|^(#\\s*%%|#\\s*\\<codecell\\>|#\\s*In\\[\\d*?\\]|#\\s*In\\[ \\])",          "jupyter.interactiveWindow.cellMarker.default": "# COMMAND ----------" }

The code can also run in VS Code (Spark session can be created — because the extension created all necessary environment variables and stored them in .databricks.env in .databricks folder.

Unfortunately, the code of databricks-connect is proprietary, and we can not tell how exactly the environment variables get evaluated.

The only thing we know, there is no magic :-)

Conclusions

We hope this article clarifies some most common confusion when developers try to develop outside of a Databricks environment.

Check out a related article about developing on Databricks!