One common explanation of the “temperature” parameter of LLMs is that it represents the “randomness” of the answer.

That’s broadly correct. Temperature is a parameter of the LLM final decoding steps, and the only one in the whole Transformer architecture that truly incorporates some randomness by design. At this stage, once the model has calculated the logits of the next token candidates, it has to map those values to an actual token from a list. Normally, LLMs perform best when they’re allowed to pick not necessarily the single best token, but instead choose at random among the N best tokens: the size of N is, more or less, what the temperature parameter represents.

Therefore, when we set the temperature to 0, the LLM must always choose the best next token, without making random choices. So, if the input is fixed and we have removed the only source of randomness in the architecture, the outputs should always be identical... right?

And yet, in practice, they often are not. Run the same prompt twice, with the same model, the same parameters, and temperature 0, and sooner or later the output will be a bit different. Not by much, usually. It may start with just one word; then the sentence takes a slightly different spin, until eventually the rest of the completion drifts away.

What’s going on?

Imperfect computations

If we pretend an LLM is just a mathematical function, temperature=0 should indeed make decoding deterministic. At each step, the model emits logits, we take the argmax token, append it to the context, and repeat. The problem is that real inference is performed with floating-point arithmetic on massively parallel hardware, usually on a server that is trying to be as fast as possible rather than mathematically pristine.

Floating-point arithmetic is only an approximation of real-number arithmetic. In particular, it is not associative: in ordinary math, (a + b) + c = a + (b + c) always holds, but with floating-point numbers those two expressions can produce slightly different results because each intermediate step is rounded. The same applies to matrix multiplications, reductions, and accumulations throughout a neural network. Change the order of operations, and you can change the last few bits of the result.

Usually, those differences are tiny and often irrelevant, but in this case they have an impact. If two candidate next tokens have very similar logits, a minute numerical difference can swap their order, and once one token changes, the next decoding step runs on a different prefix, so the divergence compounds. The sampling rule is deterministic, while the computation that produced the logits is not guaranteed to be identical across runs.

You can think of it this way: sampling determinism is not the same thing as system determinism.

It gets worse

However, this is only part of the problem. You may already be objecting that running the same matrix multiplication on a GPU with the same data repeatedly will always provide bitwise-identical results. The computations are done in floating-point arithmetic, and there are surely other jobs running on the GPU while your computer is on. So why are these calculations deterministic, while LLM sampling with temperature=0 is not?

In a recent post on Thinking Machines’s blog, Defeating Nondeterminism in LLM Inference, Horace He’s digs even deeper into the issue. It’s not merely that floating-point arithmetic is imperfect. Modern inference systems also need to batch requests together, and the result for one request can depend on the batch context in which it was executed. For a given exact batch, the forward pass may be deterministic. But from the user’s point of view, the system is still nondeterministic, because the batch itself is not stable from run to run. Your prompt may be identical, but the inputs that get batched together with yours are not.

This is also why a prompt can look stable in local testing and then become flaky in production: the model did not suddenly become more creative, it’s the system conditions that changed. temperature=0 makes only the token selection rule deterministic. It does not guarantee that the entire inference system will produce exactly the same logits every time.

Can it be fixed?

The way LLMs inference works today, especially at scale, doesn’t leave us with many options to enforce the conditions that can guarantee deterministic outputs. There are only trade-offs, and they differ quite a lot between hosted APIs and self-hosted inference.

Fixed seeds

To reduce randomness and make LLM outputs reproducible, some people recommend using a fixed seed, and indeed some providers expose one. OpenAI, for example, documents a seed parameter and says it makes a best effort to sample deterministically, while explicitly warning that determinism is not guaranteed and that backend changes can still affect outputs. Their system_fingerprint field exists precisely so you can notice when the underlying serving configuration has changed.

The problem with fixed seeds is that they help reproduce results when the temperature is above zero, not when it’s already zeroed out. That’s because a fixed seed controls the randomness of the sampling step: by setting the temperature to zero, we are already removing that source of randomness, so the net result is identical with or without a fixed seed, while every other source of nondeterminism coming from the GPU and the rest of the stack is unaffected.

So fixed seeds are worth using when you are trying to get the same results for a call with non-zero temperature, such as for tests, demos, and regression checks. But you must keep in mind that they affect only the sampler, and they won’t help you when temperature is zero.

No parallel jobs

If you self-host, one option to drastically reduce randomness is to reduce or eliminate concurrency.

This works for the simple reason that it stabilizes batching and scheduling. vLLM’s reproducibility guidance says that by default it does not guarantee reproducibility on its own. In offline mode, you should disable multiprocessing to make scheduling deterministic, while in online mode, you need batch invariance support if you want outputs that are insensitive to batching. vLLM also documents batch invariance as a distinct feature and notes that it currently depends on specific hardware support.

This means that you can pick a few different configurations, depending on your needs:

• shared online serving with dynamic batching: fastest, cheapest, least reproducible

• isolated worker / no concurrent jobs: slower, more expensive, more reproducible

• specialized batch-invariant serving paths: better reproducibility, but with hardware and feature constraints

The overall pattern is that the more you optimize for throughput, the more reproducibility suffers.

Cache responses

Caching doesn’t exactly address the reproducibility issue per se, but in many applications it’s the right level of abstraction if you want the same input to produce the same output. It’s often not only the most viable option, but also the cheapest, simplest, and fastest, unless you’re running a benchmark or an evaluation.

In practice, if you just need the same visible result for the same request, the most reliable method is not to regenerate it at all. Normalize the prompt, model ID, and relevant parameters into a cache key, store the first successful response, and serve that on subsequent identical requests. This does not make the model deterministic, of course, but it does make your application deterministic at the interface boundary, which is usually what application builders need.

Caching also has a very nice advantage over seeds and scheduler tricks: it does not depend on hidden implementation details inside the inference stack.

Of course, caching has limits. It only helps when requests repeat, and it can become awkward if tool calls, timestamps, external retrieval, or hidden context make two apparently identical requests not truly identical. Still, it is usually far more convenient than any other solution to this problem, and the only practical one for most production systems.

Conclusion

When faced with LLM nondeterminism, there’s often the reaction to treat it like a bug and to try to eliminate that. However, you should also keep in mind that LLMs were designed with a randomness factor built-in for a reason: because they perform much better when they are allowed a slight degree of nondeterminism.

I get it: nobody likes having such a huge, random black box at the core of an application’s business logic. But removing randomness from the outputs is not the right way to manage an LLM’s behavior. If you need completely deterministic output, it is better to use the LLM to design a decision tree (or a more sophisticated model, if needed) and then use that in your application.

Handling LLM outputs is rather matter of validation. Use schemas and validators so small textual drift does not break downstream code. Use evals instead of spot-checking. Cache where consistency matters, or where you need to save a few bucks. In other words, handle the randomness at the system boundary rather than trying to remove it from the model itself.

For the past two years, the default architecture for giving LLMs access to a knowledge base has been RAG with vector databases.

This architecture turned out to be very powerful, but it’s far from cheap to setup and maintain: you need the system to chunk all the documents, embed all the chunks, store them in a vector DB, retrieve them, and feed them to the model. Every new document needs to go through this pipeline before it’s usable, and changes to a document already processed means going through the vector DB and deleting all the affected chunks.

So it may come as a big surprise for many of us to learn that, as some people claims, an agent equipped with grep can find the data it’s looking for just as well.

For example, one of the many comparisons that have been done on this topic found roughly these figures when answering questions about a Django codebase:

1. Vector search over embedded chunks achieved ~60% accuracy.

2. Agentic search using tools like grep, find, and cat, where the model iteratively explores the repository, achieved ~68%.

The same test on a TypeScript/Go codebase had the two approaches both reaching around ~70%.

The difference was cost and context: vector search consumed significantly more tokens. While it arguably provided more context to the agent, it’s not clear whether the context retrieved this way was more useful, and it’s easy to find contradicting results on the Internet.

So, what’s truly going on?

If I had to summarize it in one sentence, it would be: retrieval quality depends heavily on the domain and the structure of the information.

Let’s check our assumptions

Classic retrieval steps in the RAG architecture assumes the query must be correct on the first try. Because of this, most of the effort and breakthroughs in this field focused on getting decent results for all possible queries.

Vector search shines at this. By getting the chunks of context that semantically come as close as possible to the query (or to an answer to it), embedding-based retrieval was crucial for the one-shot retrieval typical of RAG apps.

However, agents remove that constraint. An agent can try an initial search with a surely subpar query, then inspect the results, refine the query, and search again.

This iterative process dramatically reduces the weaknesses of keyword search and, in fact, leverages all its strengths: exact keyword retrieval is not an ideal task for a vector DB, because semantically similar keywords will also be included.

This does not mean vector search is useless in the age of agents.

Domain Differences

Retrieval strategies behave very differently depending on the domain. Let’s see a few examples to understand where one approach or the other shines.

Code Search

Code search is a perfect candidate for agentic keyword search, because identifiers are keywords that need exact matches in the results.

Vector search, while possible, has always been difficult to perform effectively on code. On top of that, there are tons of tools and techniques made for human coders to navigate a codebase with keywords, and agents can take advantage of those. For example, agents can use grep, AST search, symbol indexing, repository graphs, and more.

There’s also a problem with context fragmentation. Vector search returns chunks, which in the case of code search are usually a fixed number of lines or symbols. Most of the time this context is useless for the agent, because it rarely includes a full logical unit, and when it does (such as when chunking on function boundaries), it becomes much harder to retrieve, because the chunks are larger.

This means that not only is vector search less precise, but it also wastes a lot of context.

How-to Guides / Knowledge Bases / General Prose

This is the classic use case where vector search shines and keyword search is far less effective.

When your corpus of text is made of conceptual explanations, natural language queries, inconsistent terminology, and so on, semantic similarity is the most likely to bring up relevant results.

Even in this case, however, pure vector search usually gets beaten by hybrid approaches, such as running vector and keyword searches in parallel and then reranking the results.

You can find more about hybrid retrieval in this other post of mine.

Legal / Medical / Scientific Documents

These sit somewhere in between. In these documents, the terminology is specialized, wording matters, and citations and sections are important. Vector search can surface relevant passages, but precision matters more than in the previous scenario. There’s more structure than in free-form prose, and you can’t lose it during the retrieval phase.

For these kinds of documents, hybrid approaches are necessary to avoid too many false positive matches.

What should I pick?

Choosing an approach usually depends on the use cases you foresee for your agent, but in practice it’s often difficult to know beforehand what kind of documents your agent will need to sift through. Even coding agents need to search the web and read technical documentation, for example.

In these situations, it’s best to avoid flattening the decision into a “keyword vs embeddings” choice. Your agent can make use of both of them and more. For example, if your agent must be able to search anything, you may give it:

• A vector DB for static, shapeless prose, for example internal knowledge bases, static “ground truth” documents, foundational data, etc. Even searching through messages on Slack and Teams may be a good fit for a vector DB.

• Tools like grep, cat, find, etc. Let the agent leverage its coding skills for quick keyword searches across all the data. Don’t forget to make the data that’s available in the vector DB also accessible through these tools.

• A simple BM25 index that can be searched for keyword matches when the results from the command-line tools are overwhelming for the agent.

• A web search tool that the agent can use to complement its local search results, if applicable.

... and so on.

Conclusion

Vector databases are not automatically the correct architecture. Neither is grep.

Before choosing, it is worth asking:

• Is the information to search through structural or semantic?

• Do queries benefit from iteration? Will my agent be able to retry the search as many times as it wants?

• Would a simple keyword search index solve most cases, or do I need to search by meaning?

Sometimes the correct system architecture is a sophisticated hybrid embedding retriever. And sometimes it is still just grep -R "the keyword".

The only way to know for sure is, as usual, a RAG evaluation pipeline. Don’t forget to measure your outcomes!

Lately, everyone is talking about deploying AI agents, but not many ask themselves what happens once those agents are out in the world.

We are used to thinking about phishing as a human problem: a person receives an unusual message, trusts it for some reason, and gives away something sensitive. But what happens when the target is not a person, but an AI agent? Can an agent be phished? And if so, what does that actually look like in practice?

Useful agents are trusted agents

AI agents are powerful precisely because they are trusted with access to many of our most private accounts. An agent may need to read email, access calendars, browse internal documentation, inspect private GitHub repositories, review tickets, or interact with SaaS tools and APIs. In other words, the agent must have both context and capability.

That also means it becomes a security boundary.

A common but weak assumption in many deployments is that if we tell the agent that some data is confidential, it will keep that data confidential. In practice, that is not a sufficient control. Once an agent is exposed to the wrong content under the wrong conditions, secrecy instructions alone do not reliably prevent leakage. But what is the wrong content and conditions? Is browsing the web the issue? Or the ability to interact with strangers? Is my air-gapped agents running on a dedicated Mac Mini secure?

The “lethal trifecta”

A useful way to think about this is through what some researchers call the lethal trifecta. The term is not especially intuitive, but the idea is simple:

If an agent has:

1. access to private data,

2. the ability to communicate externally, and

3. exposure to untrusted content,

then you agent is vulnerable by design, and there is a path to data exfiltration.

The original definition of the “lethal trifecta” comes from Simon Willison’s blog.

The exact exploit path may be simple or more complicated. It may take one try or many. But if those three conditions are present, the question is often not whether a leak is possible, but when it will happen.

This matters because many real agents satisfy these conditions almost by default. A very small agent that can read email, browse the web, and access a secret is already vulnerable.

That does not mean every such agent will be compromised immediately. It does mean you should not assume that prompt instructions alone make it safe.

Demo setup

To make this concrete, I built a minimal demo in a controlled environment. You can find all the code I used to build this demo here.

The demo agent is implemented in n8n as a low-code workflow and it’s intentionally very simple:

• it receives chat input formatted as if it were email,

• it’s powered by a modern frontier model, specifically GPT 5.2,

• it has access to only one tool, HTTP GET, for web browsing,

• it operates in a small local environment with a fake search engine, fake documentation pages, and a fake SaaS product.

The agent’s achitecture in n8n.

Somewhere in this environment I also placed an attacker’s trap, and if the agent falls for it, we should receive the leaked credentials through a Telegram message.

The agent’s system prompt contains instructions and a few secrets, including an API key for our imaginary SaaS product. The prompt explicitly told the agent not to share those credentials with anybody and indeed, if you directly ask the agent for the API key, it refuses.

That is what many teams observe in testing, and it often creates false confidence. The agent appears aligned. It appears to understand that the credential is sensitive. But direct requests are the easy case: our demo is not about getting the model to share the credentials through an email, or some other form or prompt injection. We’re gonna demonstrate an entirely different attack surface, something much more similar to regular phishing as conducted against human targets.

A plausible support request

The interesting failure mode appears when the attacker does not ask for the secret directly. Instead, they send something that looks like a normal support or troubleshooting request:

I’m trying to call this endpoint on this SaaS API but I can’t get it to work. Can you send me a working example?

This is exactly the kind of task a helpful agent is supposed to solve. So the agent does what a helpful agent would do:

1. it searches for relevant documentation,

2. it follows documentation links,

3. it discovers API references or an OpenAPI spec,

4. it come across for a sandbox or example environment,

5. and it tries to produce a working example.

And that is where the leak happens.

In the demo, the agent found documentation that pointed to a sandbox endpoint controlled by the attacker. The agent treated that documentation as legitimate, believed the sandbox was part of the normal workflow, and tested the integration using the real API key it had been given.

The result: the attacker received the credential.

Not because the agent was asked to disclose it, but because the agent was induced to use it in the wrong place.

That distinction matters. Many defenses focus on preventing explicit disclosure. Real attacks often succeed by steering the agent into operational misuse instead.

How does the trick work?

The attack depends on a simple fact: the agent is willing to treat some external content as trustworthy enough to act on.

If attacker-controlled documentation, examples, links, or sandboxes are interpreted as valid guidance, then the agent can be manipulated into doing work on the attacker’s infrastructure. Once that happens, any credential it uses may be exposed.

Usually, a developer won’t use a real API key on a sandbox system randomly found on the web, but an agent, being more naive, will try it out.

Seen under this light, this is a phishing attack:

• The email to the agent is the lure.

• The malicious documentation is the fake login page.

• The sandbox is where the credential gets captured.

And critically, a limited toolset does not save you. Restricting an agent to HTTP GET does not eliminate prompt-injection or phishing-style risk. If the agent can fetch attacker-controlled content and then use secrets in a way that causes outbound requests, that can be enough.

In the real world

A common reaction is that this sounds artificial: surely a fake documentation page will never outrank the real docs, and an agent will be able to tell the difference, right? Surely the agent will not fall for something that naive.

That objection misses two things. First, attackers have time. They can try many variants, test against many products, and refine their lures, and with the help of modern LLMs, setting up a trap like this takes a hour at most, and can be automated to a large degree. Second, the search engine is not even necessary: an attacker can send the link directly by email, ticket, document, chat message, or issue comment. If the agent consumes the content and treats it as actionable, that may be enough.

Also, the leak only needs to happen once.

A stolen API key does not announce itself! If the attacker uses it quietly, the victim may not notice for some time. That makes one-time leakage operationally serious even if the exploit is intermittent.

What to do about it

The real problem is that there is no complete, simple fix today. If your architecture satisfies the lethal trifecta, you should assume residual exfiltration risk remains. That said, some mitigations are still worth applying to reduce your attack surface area.

1. Use disposable, low-privilege credentials

Do not give agents credentials you cannot afford to lose.

Prefer:

• narrowly scoped API keys,

• short-lived credentials,

• credentials and keys that are easy to rotate,

• strong isolation between environments,

• permissions minimized to exactly what the agent needs.

If a key leaks, recovery should be operationally manageable and should not bankrupt you.

2. Monitor and review credential use

If agents are using secrets, their activity should be observable. If your agent is exposed and has access to some sensitive credentials, you should consider setting up:

• usage logs,

• anomaly detection,

• per-agent attribution,

• alerts for unusual destinations or access patterns,

• rapid revocation workflows.

This is not perfect, but it reduces dwell time after compromise.

3. Red-team the agent continuously

Static evaluation is not enough.

When you discover a new attack pattern, test it against your own deployment. If your agent reads email, test adversarial email. If it reads docs, test malicious docs. If it uses APIs, test whether it can be tricked into authenticating to the wrong place. Then assess the damage and reinforce the prompt guardrails, tighten whitelists, but also improve your own process for recovery from the type of leak you observed, because no safeguard is 100% secure today.

Treat agent security as an ongoing adversarial exercise, not a one-time review.

4. Improve prompts

Stronger system prompts can help. You can include examples of prompt injection, tool misuse, credential theft, malicious links, and suspicious documentation patterns. However, this should be treated as one layer, not as a primary guarantee. Prompting can reduce some classes of failure, but does not remove the inherent risk.

Architectural defenses

The more serious defenses are architectural.

A promising direction is to separate the model that reads untrusted content from the model or system that has tool access and secrets. In other words, do not let the same component both interpret attacker-controlled input and directly act with privileged credentials. This kind of separation reduces the chance that malicious instructions flow directly from content ingestion into secret-bearing tool use.

One example of this broader design direction is the CaMeL approach, where responsibilities are split across components with different trust assumptions. That area is still early, and there are not yet many mature production implementations, but it points toward a more defensible model than “one agent does everything.”

Conclusion

State of the art is improving, but most current agent implementations still do not adequately account for these threats. If you are deploying agents into real workflows, especially workflows that combine private data, external communication, and untrusted content, then you need to be much more careful than most demos and product pages suggest.

The problem is not just that an agent might say the wrong thing to the wrong person. The problem is that a capable agent can be manipulated into doing the wrong thing with your secrets.

That is what phishing an AI agent looks like. If you are building or deploying agents today, assume they are vulnerable to this class of attack.

People often talk about an LLM “remembering” (or more often “forgetting”) things. But how is that possible? LLMs are stateless algorithms that don’t inherently have the ability to “remember” anything they see after their training is over. They don’t have anything like databases, caches, logs. At inference time, LLMs produce the next token based only on its trained parameters and whatever text you include in the current request.

So what is “memory” in the context of LLM inference?

The chat history

When you’re having a conversation with an LLM, the LLM does not remember what you’ve said in your previous messages. Every time it needs to generate a new token it re-reads everything that happened in the conversation so far, plus everything it has generated up to that point, to be able to decide what’s the most likely next token. LLMs don’t have any internal state: everything is recomputed from scratch for each output token.

💡 Methods exist to reduce the time complexity of LLM inference, mostly in the form of smart caching techniques (usually called prompt caching), but that’s a story for another blog post.

This means that the chat history is not part of the LLM, but it’s managed by the application built on top of it. It’s the app’s responsibility to store the chat history across turns and send it back to the LLM each time the user adds a new message to it.

The storage of the chat history is the simplest implementation of what “memory” means for an LLM. We can call it short-term memory and it allows the LLM to have a coherent conversation for many turns.

However, this approach has a limit: the length of the conversation.

The context window

LLMs can only process a fixed maximum amount of text at once. This limit is called context window and includes both the user’s input (which in turn includes all the chat history up to that point) plus the output tokens the LLM is generating. This is an unavoidable limitation of the architecture of Transformer-based LLMs (which includes all the LLMs you’re likely to ever come across).

So, what happens when the context window fills up? In short, the LLM will crash.

To prevent a hard system crash, various LLM applications handle context window overflows differently. The two most basic approaches are:

1. Hard failure (common in APIs): If you exceed the model’s context window, the request fails.

2. Truncation/sliding window (common in chat apps): The application drops older parts of the conversation so the latest turns fit. This means that for each new token you or the LLM are adding to the chat, an older token disappears from the history, and the LLM “forgets” it. In practice, during a conversation this may look like the LLM forgetting older topics of conversation, or losing sight of its original goal, or forgetting the system prompt and other custom instruction you might have given at the start of the chat.

However, both of these are just patches over the fundamental problem that LLMs can’t remember more than the content of their context window. How do we get around that to achieve long-term memory?

LLM memory is context engineering

Making LLMs able to remember very long conversations is a context engineering problem: the science of choosing what to put in the LLM’s context window at each inference pass. The context window is a limited resource, and the best LLMs applications out there usually shine due to their superior approach to context engineering. The more you can compress the right information into the smallest possible context, the faster, better and cheaper your AI system will be.

In the case of long-term memory, the core of the problem is choosing what to remember and how to make it fit into the context window. There are three common approaches: summarization, scratchpad/state, and RAG. These are not mutually exclusive, you can mix and match them as needed.

Summarization

In the case of summarization-style memory, the idea is to “compress the past” to make it fit the context window. You keep recent messages verbatim, but you also maintain a rolling summary of older conversations and/or older messages in the same conversation. When the chat gets long, you update the summary and discard raw older turns.

This is a pragmatic fit for simple chatbots: most users don’t expect perfect recall, but are happy with an LLM that sort of remembers a summary of what they talked about in the past. It’s also rather cheap and very simple to implement, which makes it a perfect fit for a quick, initial implementation.

The main issue with summarization memory is that LLMs often don’t know what details must be remembered and what can be discarded, so they’re likely to forget some important details and this might frustrate the users.

In short, summarization memory achieves something very like human memory: infinitely compressible but likely to lose details in arbitrary ways. This works for role-playing chatbots for example, but not for personal assistants that are supposed to remember everything perfectly.

Scratchpad

In order to overcome the fallacies of human memory, people use post-its and notebooks to store important details that can’t be forgotten. Turns out that LLMs can do this too! This is called scratchpad / state approach and means that the LLM is now in charge of maintaining a small, structured “state” that represents what the assistant should not forget, such as user preferences current goals, open tasks, todo lists, key decisions, definitions and terminology agreed upon, and more.

This approach can be implemented in two ways:

• by giving a scratchpad tool to the LLMs, where the model can choose to write, edit or delete its content at all times,

• by having a separate LLM regularly review the conversation and populate the scratchpad.

In either case, the scratchpad content is then added to the conversation history (for example in the system prompt or in other dedicated sections) and older conversation messages are dropped.

This approach is far more controllable than summaries, because the LLM can be instructed carefully as of what it’s critical to remember and how to save it into the scratchpad. Not only, but the users themselves can be allowed to read and edit the scratchpad to check what the LLM remembers, add more information, or even correct errors.

RAG Memory

But what if the scratchpad becomes itself huge and occupies a large share of the context window, or even overflows it? For agents that need to take on huge tasks (for example coding agents and deep research systems) the scratchpad approach might not be enough.

In this case we can start to treat memory as yet another data source and perform RAG over the scratchpad and/or the conversation history, stored in a vector DB and indexed regularly.

The advantage of RAG memory is that you can reuse all well-known patterns for RAG, with the only difference that the content to be retrieved is the chat history itself and/or the LLM’s notes.

However, RAG memory suffers from the shortcomings of retrieval: as the retrieval pipeline is never absolutely perfect, you can’t expect perfect recall. You’ll have to pay attention to the quality of the memory retrieval, evaluate it carefully and regularly, and so on. This adds a new dimension to your agent’s evaluation and in general quite a bit of complexity.

In addition, you may run into an additional problem that’s unique to RAG memory: context stuffing. Context stuffing is the presence of retrieved snippets of context that look like prompts: they can cause problems because they might confuse the LLM into following the instruction contained in the retrieved snippet instead of the user’s instruction.

While context stuffing can happen with malicious context snippets in regular RAG, it’s also very likely to happen accidentally when implementing RAG-based memory that searches directly into the chat history. This happens because all the retrieved snippets were indeed user’s prompts in the past! In this case, it’s essential to make sure that the prompt identifies clearly the retrieved snippets as context and not prompts.

Conclusion

That’s it! With any of these three approaches, your LLM-base application is now able to remember things long-term.

However, don’t forget that the moment when you add memory to your LLM powered application, you’re now storing user data, with all the problems that this brings. You will need to take care of retention, user control over the memorized data, you’ll be storing PII and secrets, and in many cases this process needs to be compliant to whatever policy for data retention you may be subject to.

Share zansara.dev

2025 was the year of LLM reasoning. Most LLM providers focused on improving the ability of their LLMs to reason, make decisions, and carry out long-horizon tasks with the least possible amount of human intervention. RAG pipelines, so hyped in the last couple of years, are now a thing of the past: the focus shifted on AI agents, a term that only recently seems to have acquired a relatively well-defined meaning:

An LLM agent runs tools in a loop to achieve a goal.

While simple, the concept at a first glance might seem to you very far from the one of RAG. But is it?

In this post I want to show you how you can extend your RAG pipelines step by step to become agents without having to throw away everything you’ve built so far. In fact, if you have a very good RAG system today, your future agents are bound to have great research skills right away. You may even find that you may be already half-way through the process of converting your pipeline into an agent without knowing it.

Let’s see how it’s done.

1. Start from basic RAG

Our starting point, what’s usually called “basic RAG” to distinguish it from more advanced RAG implementations, is a system with a retrieval step (be it vector-based, keyword-based, web search, hybrid, or anything else) that occurs every time the user sends a message to an LLM. Its architecture might look like this:

Systems with more than one retriever and/or a reranker step also fall under this category. What’s crucial to distinguish basic RAG from more “agentic” versions of it is the fact that the retrieval step runs on every user message and that the user message is fed directly to the retriever.

2. Add Query Rewrite

The first major step towards agentic behavior is the query rewrite step. RAG pipelines with query rewrite don’t send the user’s message directly to the retriever, but rewrite it to improve the outcomes of the retrieval.

Query rewrite is a bit of a double-edged sword. In some cases it may make your RAG pipeline less reliable, because the LLM may misunderstand your intent and query the retriever with an unexpected prompt. It also introduces a delay, as there is one more round-trip to the LLM to make. However, a well implemented query rewrite step has a huge impact on follow-up questions.

Think about a conversation like this:

User: What do the style guidelines say about the use of colors on our website?

Assistant: The style guidelines say that all company websites should use a specific palette made of these colors: ....

User: Why?

The first questions from the user is clear and detailed, so retrieval would probably return relevant results regardless of whether the query gets rewritten or not. However, the second question alone has far too little information to make sense on its own: sending the string “Why?” to a retriever is bound to return only garbage results, which may make the LLM respond something unexpected (and likely wrong).

In this case, query rewrite fixes the issue by expanding the “Why?” into a more reasonable query, such as “What’s the reason the company mandated a specific color palette?” or “Rationale behind the company’s brand color palette selection”. This query helps the retriever find the type of information that’s actually relevant and provide good context for the answer.

3. Optional Retrieval

Once query rewrite is in place, the next step is to give the pipeline some very basic decisional power. Specifically, I’m talking about skipping retrieval when it’s not necessary.

Think about a conversation like this:

User: What do the style guidelines say about the use of colors on our website?

Assistant: The style guidelines say that all company websites should use a specific palette made of these colors: ....

User: List the colors as a table.

In this case, the LLM needs no additional context to be able to do what the user asks: it’s actually better if the retrieval is skipped in order to save time, resources, and avoid potential failures during retrieval that might confuse it (such as the retriever bringing up irrelevant context snippets).

This means that even before query rewrite we should add another step, where the LLM gets to decide whether we should do any retrieval or not. The final architecture looks like this:

💡 Note that this is just a naive implementation. In practice, the decision of retrieving and the query rewrite may be done by the same LLM call to save time. You may also use different LLMs in parallel for different steps, leveraging smarter and more expensive LLMs for the decisional tasks and faster/cheaper ones for the query rewrite and the answer generation.

This is a critical step towards an AI agent: we are giving the LLM the power to take a decision, however simple the decision may look. This is the point where you should start to adapt your evaluation framework to measure how effective the LLM is at taking decisions, rather than its skills at interpreting the retrieved context or the effectiveness of your retrieval step alone. This is what Agent evaluation frameworks will do for you (see the bottom of the article for some suggestions).

4. The Agentic Loop

Once we have this structure in place, we’re ready to give the LLM even more autonomy by introducing an agentic loop.

Since the LLM is now able to take the decision to retrieve or not retrieve based on the chat history, how about we let the LLM also review what context snippets were returned by the retriever, and decide whether the retrieval was successful or not?

To build this agentic loop you should add a new step between the retrieval and the generation step, where the retrieved context is sent to the LLM for review. If the LLM believes the context is relevant to the question and sufficient to answer it, the LLM can decide to proceed to the answer generation. If not, the process loops back to the query rewrite stage, and the retrieval runs again with a different query in the hope that better context will be found.

The resulting architecture looks like this:

💡 Note that this is also a naive implementation. A few of these decisions can be packed together in a single pass and, again, you can use different LLMs for different tasks.

With the introduction of the agentic loop we’ve crossed the boundary of what constitutes an AI Agent, even though it’s still a very simple one. The LLM is now in charge of deciding when the retrieval is good enough, and it can try as many times as it wants (up to a threshold of your choosing) until it’s satisfied with the outcome.

If your retrieval step is well done and effective, this whole architecture may sound pointless. The LLM can hardly get better results by trying again if retrieval is already optimized and query rewriting is not making mistakes, so what’s the point? In this case, the introduction of the agentic loop can be seen just as a necessary stepping stone towards the next upgrade: transforming retrieval into a tool.

5. Retrieval as a Tool

In many advanced RAG pipelines, retrieval of context and tool usage is seen as two very different operations. RAG is usually always on, highly custom, etc. while tools tend to be very small and simple, rarely called by the LLM, and sometimes implemented on standardized protocols like MCP.

This distinction is arbitrary and simply due to historical baggage. Retrieval can be a tool, so it’s best to treat it like one!

Once you adopt this mindset, you’ll see that the hints were there all along:

1. We made retrieval optional, so the LLM can choose to either call it or not - like every other tool

2. Query rewrite is the LLM choosing what input to provide to the retriever - as it does when it decides to call any other tool

3. The retriever returns output that goes into the chat history to be used for the answer’s generation - like the output of all other tools.

Transforming retrieval into a tool simplifies our architecture drastically and moves us fully into AI Agent territory:

As you can see:

1. The decision step is now part of the LLM’s answer generation, which can call it as many times as it wants thanks to the tool calling loop

2. The query rewrite comes for free as the LLM invokes the retrieval tool

3. The retriever’s output goes into the chat history to be used to answer the user’s request

At this point it’s time to address a common concern. You may have heard elsewhere that implementing retrieval as a tool makes the LLM “forget” to retrieve context when it should rather do it, so the effectiveness of your RAG worsens. This was very real a couple of years ago, but in my experience it’s no longer relevant: modern LLMs are now trained to reach for tools all the time, so this problem has largely disappeared.

6. Add more tools

Congratulations! At this point you can call your system a true AI Agent. However, an agent with only a retrieval tool has limited use. It’s time to add other tools!

To begin with, if your retrieval pipeline has a lot of moving parts (hybrid retriever, web search, image search, SQL queries, etc...) you can consider separating each of them into separate search tools for the LLM to use, or to expose more parameters to let the LLM customize the output mix.

Once that’s done, adding other tools is trivial on a technical level, especially with protocols such as MCP. Using popular, open source MCPs may let you simplify your retrieval tool drastically: for example by leveraging GitHub’s MCP instead of doing code search yourself, or Atlassian’s MCPs instead of custom Jira/Confluence/BitBucket integrations, and so on.

However, keep in mind that adding too many tools and MCPs can overwhelm the LLM. You should carefully select which tools can expand the most your LLM’s ability to solve your user’s problems. For example, a GitHub MCP is irrelevant if only very few of your users are developers, and an image generation tool is useless if you’re serving only developers. It’s easy to overdo it, so make sure to review regularly the tools you make available to your LLM and add/remove them as necessary.

And in the rare case in which you actually need a lot of tools, consider letting the user plug them in as needed (like the ChatGPT UI does), or adopt a more sophisticated tool calling approach to make sure to manage the context window effectively.

Conclusion

That’s it! You successfully transformed your RAG pipeline into a simple AI Agent. From here you can expand further by implementing planning steps, sub-agents, and more.

However, before going further you should remember that your retrieval-oriented metrics now are not sufficient anymore to evaluate the decision making skills of your system. If you’ve been using a RAG-only eval framework such as RAGAS it’s now a good time to move on to a more general-purpose or agent-oriented eval framework, such as DeepEval, Galileo, Arize.ai or any other AI Agent framework of your choice.

Last but not least: if you want to see this entire process implemented in code, don’t miss my workshop at the virtual Agentic AI Summit on the 21st of January, 2026! I’ll be walking you through the entire process and show you some additional implementation details. See you there!

Nearly all popular LLMs share the same internal structure: they are decoder-only Transformers. However, they are not completely identical: in order to speed up training, increase intelligence or improve inference speed and cost, this base template is sometimes modified a bit.

One popular variant is the so-called MoE (Mixture of Experts) architecture: a neural network design that divides the model into multiple independent sub-networks called “experts”. For each input, a routing algorithm (also called gating network) determines which experts to activate, so only a subset of the model’s parameters is used during each inference pass. This leads to efficient scaling: models can grow significantly in parameter size without a proportional increase in computational resources per token or query. In short, it enables large models to perform as quickly as smaller ones without sacrificing accuracy.

But what are these expert networks, and how are they built? One common misconception is that the “experts” of MoE are specialized in a well defined, recognizable type of task: that the model includes a “math expert”, a “poetry expert”, and so on​. The query would then be routed to the appropriate expert after the type of request is classified.

However, this is not the case. Let’s figure out how it works under the hood.

The MoE architecture

In order to understand MoE, you should first be familiar with the basic architecture of decoder-only Transformers. If the diagram below is not familiar to you, have a look at this detailed description before diving in.

The main change made by a MoE over the decoder-only transformer architecture is within the feed-forward component of the transformer block. In the standard, non MoE architecture, the tokens pass one by one through a have a single feed-forward neural network. In a MoE instead, at this stage there are many feed-forward networks, each with their own weights: they are the “experts”.

This means that to create an MoE LLM we first need to convert the transformer’s feed-forward layers to these expert layers. Their internal structure is the same as the original, single network, but copied a few times, with the addition of a routing algorithm to select the expert to use for each input token to process.

The core of a routing algorithm is rather simple as well. First the token’s embedding passes through a linear transformation (such as a fully connected layer) that outputs a vector as long as the number of experts we have in our system. Then, a softmax is applied and the top-k experts are selected. After the experts produce output, their results are then averaged (using their initial score as weight) and sent to the next decode layer.

Keep in mind that this is a simplification of the actual routing mechanism of real MoE models. If implemented as described here, through the training phase you would observe a routing collapse: the routing network would learn to send all tokens to the same expert all the time, reducing your MoE model back to the equivalent of a regular decoder-only Transformer. To make the network learn to distribute the tokens in a more balanced fashion, you would need to add auxiliary loss functions that make the routing network learn to load balance the experts properly. For more details on this process (and much more on MoE in general) see this detailed overview.

So experts never specialize?

Yes and no. On the OpenMoE paper, the authors investigated in detail whether experts do specialize in any recognizable domain, and they observed interesting results. In their case, experts do not tend to specialize in any particular domain; however, there is some level of expert specialization across natural languages and specific tasks.

According to the authors, this specialization is due to the same tokens being sent to the same expert every time, regardless of the context in which it is used. Given that different languages use a very different set of tokens, it’s natural to see this sort of specialization emerging, and the same can be said of specific tasks, where the jargon and the word frequency changes strongly. The paper defines this behavior as “Context-Independent Specialization”.

It’s important to stress again that whether this specialization occurs, and on which dimensions, is irrelevant to the effectiveness of this architecture. The core advantage of MoE is not the presence of recognizable experts, but on the sparsity it introduces: with MoE you can scale up the parameters count without slowing down the inference speed of the resulting model, because not all weights will be used for all tokens.

Conclusion

The term “Mixture of Experts” can easily bring the wrong image into the mind of people unaccustomed with how neural networks, and Transformers in general, work internally. When discussing this type of models, I often find important to stress the difference between how the term “expert” is intended by a non technical audience and what it means in this context.

If you want to learn more about MoEs and how they’re implemented in practice, I recommend this this very detailed article by Cameron Wolfe, where he dissects the architecture in far more detail and adds plenty of examples and references to dig further.

It has been a long time since TF-IDF or even BM25 were the state of the art for information retrieval. These days the baseline has moved to embedding similarity search, where each unit of information, be it a sentence, a paragraph or a page is first encoded in an embedding and then compared with the embedding of the user’s query.

From this baseline there are often two pieces of advice to help you increase the performance of your search system: one is to go the deep end with the embedding approach and consider a reranker, finetune your embedding model, and so on. The other, usually called hybrid retrieval or hybrid search, is to bring back good old keyword search algorithms and use them to complement your results. Often the best scenario is to use both of these enhancements, which nicely complement each other.

But why would this arrangement help improve the results? Isn’t embedding search strictly superior to keyword-based retrieval algorithms?

Semantic vs Lexical

When you embed a sentence, the resulting embedding encodes its meaning, not its exact phrasing. That’s their strength! But it can often be a limitation as well.

For example a semantic model can understand that “latest iPhone” is similar to “iPhone 17 Pro Max”, which is great if the first sentence is a query and the second the search result. But a semantic model will also say that “iPhone 17 Pro Max” and “iPhone 11 Pro Max” are very similar, which is not great if the first sentence is a query and the second a search result.

In short, semantic similarity is great if you are starting from a generic query and you want a set precise result all matching the generic description, or if you start from a general question and want to retrieve all very particular results that fall under the same general concept. For “latest iPhone”, “iPhone 17 Pro Max”, “iPhone 17 Pro” and ideally “iPhone Air” and are all valid search results.

On the other hand, lexical similarity is what allows your system to retrieve extremely precise results in response to a very specific query. “latest iPhone” will return garbage results with a lexical algorithm such as BM25 (essentially any iPhone would match), but if the search string is “iPhone 17 Plus Max”, BM25 will return the best results.

To visualize it better, here’s the expected results for each of the two queries in a dataset of iPhone names:

As you can see, the problem is that neither of the two approaches works best with both types of queries: each has its strong pros and cons and works best only on a subset of the questions your system may receive.

So why not using them both?

Combining them

A hybrid search system is simply a system that does the same search twice: once with a keyword algorithm such as BM25, and once with vector search. But how to merge the two lists of results?

The scores the documents come with are deeply incomparable. BM25 scores depends on terms frequency and keyword matching, and are not bound to any range. On the contrary, cosine similarity usually clusters between 0.5 and 0.9, which gets even narrower if the sequences are longer.

That’s where reciprocal rank fusion (RRF) comes in. RRF is incredibly simple and boils down to this formula: score(d) = sum( 1/(k + rank_method_i(d)) ) . As you can see it works on the ranks, not scores, so it’s robust against scale differences and requires no normalization. Platforms like Elastic and Pinecone use it for production hybrid search due to its simplicity and reliability. Being so simple, the additional latency is negligible, which makes it suitable for real-time usecases.

Or, if you’re less concerned about latency, you can consider adding a reranker.

Having two independent and complementary search techniques is the reason why adding a reranker to your hybrid pipeline is so effective. By using these two wildly different methods, it’s not obvious whether even the rankings are comparable. Rerankers can have a more careful look at the retrieved documents and make sure the most relevant documents are to the top of the pile, allowing you to cut away the least relevant ones.

Conclusion

Hybrid search isn’t a patch for outdated systems, but a default strategy for any high-quality retrieval engine. Dense embeddings bring rich contextual understanding, while sparse retrieval ensures accuracy for unique identifiers, numeric codes, acronyms, or exact strings that embeddings gloss over. In a world where search systems must serve both humans and machine agents, hybrid search is the recall multiplier that guarantees we get both meaning and precision.

With time, a veritable zoo of optimization strategies arose to get around this problem, and making sense of which optimizations can be applied to which model can be a challenge in itself. Fortunately a very comprehensive survey on KV caching recently collected all techniques that make up the state of the art in this field, giving practitioners a handy starting point to understand this field. The amount of techniques reviewed is staggering, so we’re going to need more than one post to go through the most interesting approaches and compare them.

For now, let’s see how we can start to make sense of things.

The challenges

Most of the techniques we’re going to see address one or more of these issues.

• Cache Eviction: Determining which items to evict when the cache reaches its capacity. Popular policies like Least Recently Used (LRU) or Least Frequently Used (LFU) do not always align with LLM usage patterns, leading to suboptimal performance.

• Memory Management: The memory required for the KV cache grows linearly with both the input length and the number of layers, which can quickly exceed the hardware memory limits. It’s possible to overcome such limits by distributing the storage of this cache across different types of storage hardware (e.g., GPU, CPU or external memory), but this brings its own set of challenges.

• Latency: Accessing and updating the cache at each decoding step can introduce latency.

• Compression: Compressing the KV cache can reduce memory usage but may degrade model performance if key information is lost.

• Dynamic Workloads: Handling dynamic and unpredictable workloads, where access patterns and data requirements frequently change, requires adaptive caching strategies that can respond in real time.

• Distributed Coordination: In distributed KV caches, maintaining coordination across multiple nodes to ensure consistency, fault tolerance, and efficient resource usage adds significant complexity.

A taxonomy

In order to make sense of the vast amount of known techniques, the authors categorized them into a comprehensive taxonomy.

Source

It starts with three major categories:

• Token-Level Optimization: improving KV cache management efficiency by focusing on the fine-grained selection, organization, and compression at the token level. These techniques can be applied to any model, as they require no architectural changes.

• Model-level Optimization: designing an efficient model structure to optimize KV cache management. These optimizations are strictly model-dependent, because they’re backed into the model’s architecture.

• System-level Optimization: optimizing the KV Cache management through techniques closer to the OS and/or the hardware. These techniques may require specialized hardware to implement, so they’re not at everyone’s reach.

Token-Level Optimizations

Token-level optimizations are the most readily accessible to most developers, as they require no dedicated support from the LLM and no specialized hardware. Therefore, these are usually the most interesting. In this category we find:

• KV cache selection: focuses on prioritizing and storing only the most relevant tokens.

• KV cache budget allocation: dynamically distributes memory resources across tokens to ensure efficient cache utilization under limited memory.

• KV cache merging: reduces redundancy by combining similar or overlapping KV pairs.

• KV cache quantization: minimizes the memory footprint by reducing the precision of cached KV pairs.

• KV cache low-rank decomposition: uses low-rank decomposition techniques to reduce cache size.

You can find a deep dive into this category in its own post.

Model-Level Optimizations

Model-level optimizations, as the name says, are baked into the model’s architecture and therefore are either not applicable or always present in the models you’re running. These optimizations are usually interesting for people that design their own model architecture and train them, rather than developers that work with off-the-shelf models. In this category we find:

• Attention grouping and sharing methods: examine the redundant functionality of keys and values and group and share KV cache within or across transformer layers.

• Architecture alterations: emerge to design new attention mechanisms or construct extrinsic modules for KV optimization.

• Non-transformer architectures: architectures that adopt other memory-efficient designs like recurrent neural networks to optimize the KV cache in traditional transformers.

You can find a deep dive into this category in its own post.

System-level Optimizations

These optimizations work across the stack to provide the best possible support to the LLM’s inference, and they’re sometimes baked into the inference engine, such as vLLM’s PagedAttention. They occasionally require dedicated hardware and OS optimizations, so they’re not always readily available for everyday experimentation. They include:

• Memory management: focuses on architectural innovations like virtual memory adaptation, intelligent prefix sharing, and layer-aware resource allocation.

• Scheduling: addresses diverse optimization goals through prefix-aware methods for maximizing cache reuse, preemptive techniques for fair context switching, and layer-specific mechanisms for fine-grained cache control.

• Hardware acceleration: including single/multi-GPU, I/O-based solutions, heterogeneous computing and SSD-based solutions.

You can find a deep dive into this category in its own post.

Conclusion

KV cache optimization is still an open research area, with new techniques and improvements being published regularly. A good overview of what types of optimizations exist can help you make sense of the zoo of acronyms and claims being made about them, and give you the foundations you need to understand if a particular technique is relevant for your situation.

To learn more about each specific category or technique, it’s a good idea to check out the survey, where you can find more detailed breakdowns and comparisons together with links to the original articles where each technique was introduces.

Happy reading!

In the previous post we saw what is prompt caching, what parts of the prompts is useful to cache, and explained at a high level why it’s so effective. In this post I want to go one step further and explain how in practice inference engines cache prompt prefixes. How can you take a complex system like an LLM, cache some of its computations mid-prompt, and reload them?

Let’s find out.

Disclaimer: to avoid overly complex and specific diagrams, the size of the vector and matrices shown is not accurate neither in size nor in shape. Check the links at the bottom of the post for more detailed resources with more accurate diagrams.

LLMs are autoregressive

Large Language Models are built on the Transformer architecture: a neural network design that excels at processing sequence data. Explaining the whole structure of a Transformer goes beyond the scope of this small post: if you’re interested in the details, head to this amazing writeup by Jay Alammar about Transformers, or this one about GPT-2 if you’re familiar with Transformers but you want to learn more about the decoder-only architecture (which includes all current LLMs).

The point that interests us is that according to the original implementation, during inference the LLM generate text one token at a time in an autoregressive fashion, meaning each new token is predicted based on all previously generated tokens. After producing (or “decoding”) a token, that token is appended to the input sequence and the model computes everything all over again to generate the next one. This loop continues until a stopping condition is reached (such as an end-of-sequence token or a length limit).

In an autoregressive system, the output is generated token by token by appending the previous pass’ output to its input and recomputing everything again. Starting from the token “This”, the LLM produces “is” as output. Then the output is concatenated to the input in the string “This is”, which is fed again to the LLM to produce “a”, and so on until an [END] token is generated. That halts the loop.

This iterative process is very computationally expensive (quadratic time complexity in the number of tokens, so O(n^2) where n is the number of tokens), and the impact is felt especially for long sequences, because each step must account for an ever-growing history of generated tokens.

Simplified view of the increasing computation load. At each pass, the increasing length of the input sentence translated into larger matrices to be handled during inference, where each row corresponds to one input token. This means more computations and, in turn, slower inference.

However, there seems to be an evident chance for optimization here. If we could store the internal state of the LLM after each token’s generation and reuse it at the next step, we could save a lot of repeated computations.

If we could somehow reuse part of the computations we did during earlier passes and only process new information as it arrives, not only the computation speed will increase dramatically, but it will stay constant during the process instead of slowing down as more tokens are generated.

This is not only true during a single request (because we won’t be recomputing the whole state from the start of the message for every new token we’re generating), but also across requests in the same chat (by storing the state at the end of the last assistant token) and across different chats as well (by storing the state of shared prefixes such as system prompts).

Example of prefix caching in different scenarios (gray text is caches, black is processed). By caching the system prompt, its cache can be reused with every new chat. By also caching by longest prefix, the prompts may occasionally match across chats, although it depends heavily on your applications. In any case, caching the chat as it progresses keeps the number of new tokens to process during the chat to one, making inference much faster.

But how can it be done? What exactly do we need to cache? To understand this we need to go one step deeper.

The inference process

At a high level, the inference process of a modern decoder-only Transformer such as a GPT works as follows.

• Tokenization: The chat history is broken down into tokens by a tokenizer. This is a fast, deterministic process that transforms a single string into a list of sub-word fragments (the tokens) plus a bunch of signalling tokens (to delimit messages, to signal the end of the message, to distinguish different types of input or output tokens such as thinking tokens, function calls, system prompts, etc)

• Embedding: the tokenized text passes through an embedding step, where each token is translated into an embedding (a 1-dimensional vector) using a lookup table. At this point, our input text has become a matrix of values with as many rows as tokens, and a fixed number of columns that depends on the LLM.

• Decoding: this matrix is passed through a series of 12 identical decoding steps. Each of these blocks outputs a matrix of the same shape and size of the original one, but with updated contents. These steps are “reading” the prompt and accumulating information to select the next best token to generate.that is passed as input to the next step

• Output: After the last decoding step, a final linear output layer projects the matrix into an output vector. Its values are multiplied by the lookup table we used during the embedding step: this way we obtain a list of values that represents the probability of each token to be the “correct” next token.

• Sampling: From this list, one of the top-k best tokens is selected as the next token, gets added to the chat history, and the loop restarts.

• End token: the decoding stops when the LLM picks an END token or some other condition is met (for example, max output length).

Simplified representation of the inference steps needed for an LLM to generate each output token. The most complex by far is the decoding step, which we are going to analyze in more detail.

As you can see from this breakdown, the LLM computes its internal representation of the chat history through its decoding steps, and recomputes such representation for all tokens every time we want to generate a new one. So let’s zoom in even more and check what’s going on inside these decoding steps.

The decoding step

LLMs may have a variable number of decoding steps (although it’s often 12), but they are all identical, except for the weights they contain. This means that we can look into one and then keep in mind that the same identical process is repeated several times.

Each decoding step contains two parts:

• a multi-headed, masked self-attention layer

• a feed-forward layer

The first layer, the multi headed masked self attention, sound quite complicated. To make things easier, let’s break it down into smaller concepts.

Attention is the foundation of the Transformers’ incredible text understanding skills and can be roughly summarized as a technique that shows the model which tokens are the most relevant to the token we’re processing right now.

Self attention means that the tokens we’re looking at belong to the same sentence we’re processing (which is not the case, for example, during translation tasks where we have a source sentence and a translation).

Masked self attention means that we’re only looking at tokens that precede the one we’re processing (which is not the case, for example, in encoder models such as BERT that encode the whole sentence at once).

Multi-headed attention means that the same operation is performed several time with slightly different parameters. Each set of parameters is called an attention head.

To understand what attention does, let’s take the sentence “I like apples because they’re sweet”. When processing the token “they”, the masked self-attention layer will give a high score to “apples”, because that’s what “they” refers to. Keep in mind that “sweet” will not be considered while processing “they”, because masked self-attention only includes tokens that precede the token in question.

A simplified visualization of a masked self-attention head. For each token, the attention calculations will assign a score to each preceding token. The score will be higher for all preceding tokens that have something to do with the current one, highlighting semantic relationships.

The Q/K/V Matrices

Let’s now look at how is this score calculated. Self-attention is implemented as a series of matrix multiplications that involves three matrices.

• Q (Query matrix): The query is a representation of the tokens we are “paying attention to” (for example, “they”. In practice all tokens will be computed at the same time, so we will be dealing with a Q matrix).

• K (Key matrix): Key vectors are like labels for all the other preceding tokens in the input. They’re what we match against in our search for relevant tokens (for example “I”, “like”, “apples”, etc ). Each token will only see the keys of tokens that precede it, so the query of “they” will not be multiplied with the key for “sweet”.

• V (Value matrix): Value vectors are actual token representations. Once we’ve scored how relevant each token is, these are the values we add up to represent the token we’re paying attention to. In our example, this means that the vector for “they” will be computed as a weighted average of all the previous tokens (“I”, “like”, “apples”, “because”), but “apples” will be weighted much higher than any other, so the end result for the token “they” will be very close to the value vector for “apples”.

These Q/K/V matrices are computed by multiplying the input of the decoding layer by three matrices (Wq, Wk and Wv) whose values are computed during training and constitute many of the LLM’s parameters. These three matrices are addressed together as an attention head, as we mentioned earlier. Modern LLMs usually include several attention heads for each step, so you’ll have several different matrices in each decoding step (and that’s why they’re said to use multi-headed attention).

Simplified view of the Q/K/V matrices in a single self-attention head. The matrices go through a few more steps (softmax, regularization etc) which are not depicted here.

This process of computing the output vector for each token is called scaled dot-product attention and, as we mentioned earlier, happens in every attention head of every decoding step. In summary, keys (K) and values (V) are the transformed representations of each preceding token that are used to compute attention, and they enable each token to gather information from the rest of the sequence by matching queries to keys and aggregating values.

Let’s pay close attention to these computations. We know that LLMs generate output one token at a time. This means that the LLM will recompute the K-V values for the tokens of the prompt over and over again for each new output token it generates. If you have already generated, say, 100 tokens of output, producing the 101st token requires recomputing a forward pass over all 100 tokens. A naive implementation would repeatedly recalculate a lot of the same intermediate results for the older tokens at every step of generation.

Detail of the Q/K multiplication. As you can see, the content of the QK matrix is essentially the same at all steps, except for the last row. This means that as soon as we accumulate a few input tokens, most of the QK matrix will be nearly identical every time. Something very similar happens for the final QKV matrix.

By the third generation step (three tokens in context), the model computes six attention scores (a 3×3 / 2 matrix); many of these correspond to interactions that were already computed in earlier steps. For example, the attention of token “I” with itself was computed in the first step, yet the naive approach computes it again when processing the sequence “I like” and “I like apples” and so on. In fact, by the time the sentence is complete, the majority of the query-key pairs being calculated are repeats of prior computations. This redundancy makes inference much slower as the sequence length grows: the model wastes time recalculating attention contributions for tokens that haven’t changed.

Clearly we want to avoid recomputing things like the key and value vectors for past tokens at every step. That’s exactly what KV caching achieves.

The KV Cache

KV caching is an optimization that saves the key and value tensors from previous tokens so that the model doesn’t need to recompute them for each new token. The idea is straightforward: as the model generates tokens one by one, we store the keys and values produced at each layer for each token in a cache (which is just a reserved chunk of memory, typically in GPU RAM for speed). When the model is about to generate the next token, instead of recomputing all keys and values from scratch for the entire sequence, it retrieves the already-computed keys and values for the past tokens from this cache, and only computes the new token’s keys and values.

In essence, with KV caching the transformer’s attention in each layer will take the new token’s query and concatenate it with the cached keys of prior tokens, then do the same for values, and move on immediately. The result is that each generation step’s workload is greatly reduced: the model focuses on what’s new instead of re-hashing the entire context every time.

Implementation

Modern libraries implement KV caching under the hood by carrying a “past key values” or similar object through successive generation calls. For example, the Hugging Face Transformers library’s generate function uses a use_cache flag that is True by default, meaning it will automatically store and reuse past keys/values between decoding steps. Conceptually, you can imagine that after the first forward pass on the prompt, the model keeps all the K and V tensors. When generating the next token, it feeds only the new token through each layer along with the cached K and V from previous tokens, to compute the next output efficiently.

In summary, KV caching transforms the workload of each generation step. Without caching, each step repeats the full attention computation over the entire context. With caching, each step adds only the computations for the new token and the necessary interactions with prior tokens. This makes the per-step cost roughly constant. The longer the generation goes on, the more time is saved relative to the naive approach. KV caching is thus a time-memory trade-off: we trade some memory to store the cache in order to save a lot of compute time on each step.

Limitations

It’s important to note that KV caching only applies in auto-regressive decoder models (where the output is generated sequentially). Models like BERT that process entire sequences in one go (and are not generative) do not use KV caching, since they don’t generate token-by-token or reuse past internal states. But for any generative LLM built on a decoder-only Transformer architecture, KV caching is a standard technique to speed up inference.

It’s worth noting that the KV cache needs to be managed just like every other type of cache. We’re going to analyze some ways to handle this cache effectively in another post.

Conclusion

The takeaway is clear: always leverage KV caching for autoregressive LLM inference (and practically all libraries do this for you) unless you have a very specific reason not to. It will make your LLM deployments run faster and more efficiently.

KV caching exemplifies how understanding the internals of transformer models can lead to substantial engineering improvements. By recognizing that keys and values of the attention mechanism can be reused across time steps, we unlock a simple yet powerful optimization. This ensures that even as our LLMs get larger and our prompts get longer, we can keep inference running quickly, delivering the snappy responses users expect from AI-driven applications.

Learn more

Here are some useful resources I used to write this post:

• The Illustrated Transformer by Jay Alammar

• The illustrated GPT-2 by Jay Alammar

• Latency optimization tips by OpenAI

• KV Caching explained by João Lages

• KV Caching by Neptune.ai

• Build a Large Language Model (from scratch) by Sebastian Raschka

A common piece of advice to improve speed and reduce cost of inference in LLMs is to use prompt caching. However, it’s often not clear what this means. What exactly is cached? When and why the improvements are really impactful? Understanding prompt caching starts with a deeper awareness of how computation and costs scale with large contexts.

LLMS are stateless

Each time an LLM processes input, it handles every token of the provided context. LLMs are stateless: this means that for every new message added to an existing chat, your application needs to submit the whole history which could include system prompts, documents, examples, and all the chat history. The model recomputes all of those tokens each time. This is a massive inefficiency. For example, with an input cost around $1 per 1 million tokens, sending 100,000 tokens across 1,000 requests would cost approximately $100, while about 95% of those tokens remain unchanged across requests. In essence, a large portion of computation is wasted on repeatedly processing information that never changes: the message history.

Stateless vs stateful design

Naive API implementations that omit caching force the model to process the entire context anew each time. This “stateless” method is simpler to implement, but wastefully expensive. The system pays repeatedly to recompute static context, which could otherwise be reused.

In contrast, with a stateful cache strategy, the system stores parts of the context and only processes new inputs (queries). Consider the following case:

• the system prompt is 10,000 tokens long

• each user message is about 100 tokens

• each assistant response is about 1000 tokens

In both cases, the first request processes 10,100 tokens (1 system prompt + 1 user message). On the second message, a stateless request (no caching) needs to process 11,200 tokens (1 system prompt + first user message + first assistant response + the next user message) while a stateful one can first load the cache and then process only 1100 new tokens (the assistant response + the new user message). That’s an order of magnitude less tokens! On top of that, as the chat continues, a stateful app will always need to only process the next new 1100 tokens, while the stateless version will process a chat history that grows by 1100 every time. For example, by the 10th request, with caching you need to process 1100 tokens, while without you need to deal with 20,000! (10,000 system prompt tokens + 9,000 assistant reply tokens + 1000 user message tokens).

Here’s a recap to highlight the difference:

While cache warm-up is not free, it can make a huge difference in the latency of your responses and, if you’re paying by the output token, reduce the costs by orders of magnitude.

Cache Hierarchies

Caching’s benefits come with architectural tradeoffs. Stateless designs are straightforward and predictably expensive: every token is always processed. Caching drastically reduces costs by reusing prior computation, but requires complexity in cache management, such as:

• Cache invalidation: deciding how and when to refresh cached segments.

• Cache misses: when requested information isn’t in the cache, leading to full recomputation and latency spikes.

Because of these challenges, a single monolithic cache usually not enough to see many benefits. The most effective solution is a hierarchical cache strategy.

Effective prompt caching leverages multiple layers with varied lifetimes and hit rates:

• L1: System Prompt (e.g., 5,000 tokens): it rarely changes, so it has the best hit rate. In most chat you’ll at least hit this cache.

• L2: System Prompt + Examples and Tools (e.g., +20,000 tokens): may change per task, so it can has a lower hit rate than the system prompt, but eventually it depends completely on your application type. Agentic apps that make heavy use of tools benefit the most from caching them, as they follow the system prompt and might not depend at all from the user query or the agent’s decisions.

• L3: System Prompt + Examples and Tools + Documents (e.g., +50,000 tokens): if you’re working with documents, caching any initial retrieval can help too. These documents are likely to change per user and/or per session, so it has a moderate/low hit rate. However, the size of these chunks usually makes it worth it if you have some spare capacity or a small and static knowledge base to retrieve from.

A layered approach like balances freshness and reuse, optimizing both cost and performance.

Automatic prefix caching

If you’re using a modern inference engine, prompt caching can also be done through automatic prefix caching, where the engine itself takes the responsibility to identify and cache frequently used prefixes. Here you can find more details about the availability of this feature in vLLM, SDLang and llama.cpp, but there are many other engines supporting it.

A feature comparison across inference engines from this May 2025 review.

Semantic caching

In extreme cases where cost, load or latency must be reduced to the maximum, semantic caching can also be employed. Semantic caching allows you to cache also the user queries and the assistant responses by keeping a registry of already processes user queries and performing a semantic search step between the new query and the cached ones. If a match is found, instead of invoking the LLM to generate a new answer, the cached reply is sent to the user immediately.

Semantic caching however has several disadvantages that makes it worthwhile only in rare situations:

• Access control. Caching must be done per user if each user has access to a different set of resources, to avoid accidental sharing of data and/or resources across users.

• Very high similarity needed: In order the the reply to be relevant, the semantic similarity between the two must be extremely high, or you risk that the answer returned to the user won’t match their question. Semantic similarity tends to overlook details which are often very important to an accurate reply: for example, “What’s the sum of these numbers: 1,2,3,4,5,6,7?” and “What’s the sum of these numbers: 1,2,3,4,5,6,7,8?” will have an extremely high similarity, but returning the response of the first to the second would not be a good idea.

• Language management: what to do when the exact same question is asked in two different languages? Semantic similarity may be perfect if your embedder is multilingual, but the user won’t be pleased to receive a cached answer in a language different from their own.

Such constraints make cache misses extremely frequent, which defies the point of keeping a cache and simply adds complexity and latency to the system instead of reducing it. The similarity pitfalls introduces also nasty accuracy problems.

In my personal experience, semantic caching is only useful for extremely high volume, low cost, public facing interfaces where accuracy is not critical. A perfect example could be a virtual assistant for anonymous customer support, or a helper bot for a software’s documentation search. In any case, you usually need additional checks on the output in order to trust such a system.

Conclusion

Prompt caching is not just about cutting costs or speeding things up: it is a necessary architectural approach that addresses the quadratic computational cost inherent in large-context LLM processing. Without it, your backend will repeatedly recompute largely static information, wasting resources and imposing latency penalties that impact your user’s experience. By adopting hierarchical, stateful caching and carefully designing prompts, you can reduce token processing costs and response speed by orders of magnitude, which is key for building sustainable, high-performance applications.

Retrieval-Augmented Generation (RAG) systems are essential to connect large language models with external knowledge sources. While in theory the retrieval step is enough to gather documents that are relevant to the user’s request, it’s often recommended to add an additional ranking step, the reranking, to further filter the results.

But why do we need rerankers? Isn’t semantic search good enough? The answer lies in understanding the limitations of traditional embedding-based retrieval.

Bi-encoders vs Cross-encoders

At the heart of modern, scalable semantic search systems lies the bi-encoder model. This architecture creates independent vector representations for the query and the document; relevance is then computed through a similarity measure like the dot product or cosine similarity between those vectors.

This design scales well: You can precompute document embeddings, store them in your vector DB, and compare any incoming query against millions of documents very efficiently. However, this convenience comes at a cost: the system never truly reads the document in the context of the query. There’s no token-level interaction between the query and document embedding to judge whether the document actually answers the question or it simply happen to be talking about the same topic, and therefore semantically similar.

For example, the query “How to protect my application from DDOS attacks?” may be semantically close to the statement “You should always take steps to protect your systems from DDOS attacks”, but the statement does not contain the answer to the question. Without reranking, embedding-based retrieval systems often perform well at recall but poorly at precision.

Cross-encoders remedy the limitations of bi-encoders by encoding the query and document together, typically separated by a special token (like [SEP]) using an encoder-only Transformer such as BERT. Then they include an additional fully connected layer that acts as a classifier, learning fine-grained token-level interactions that capture query/document word alignments, answer containment (i.e., does this passage actually answer the question?) and overall contextual relevance.

This difference (from separate encodings to a joint representation) gives cross-encoders their power but also their cost. Since relevance depends on the specific query, you can’t precompute the document embeddings: in fact, the concept of “query embedding” and “document embeddings” disappears. Every query-document pair requires a fresh forward pass through the whole model, which can be prohibitively expensive on a large corpus.

To each their place

No production system can afford to run interaction-rich models such as cross-encoders on millions of documents per query. Therefore, the two-stage retrieval pipeline remains the industry standard:

1. Semantic Search (Bi-Encoder) – Quickly narrows a massive corpus (e.g., millions of document chunks) down to a small candidate set (e.g., top 100 chunks). Bi-encoders can be built with any embedding model: popular closed source embedders include OpenAI’s, Voyage.ai, Cohere’s, Gemini and more, while on the open-source front you can find BGE embedders, Mistral’s models, Jina.ai, Gemma, IBM Granite, and more.

2. Reranking (Cross-Encoder) – Evaluates those top 100 candidates more deeply by jointly encoding the query and document. A popular closed source choice for reranking models is Cohere’s, while on the open source front you can find several Qwen-based rerankers, Jina.ai models, IBM’s Granite rerankers, BGE rerankers, and many more.

Making Reranking Practical

Even in this two-tiered system, reranking may turn out to be too expensive for your latency constrains, but several engineering and modeling strategies have emerged to make it viable in production. Let’s break down a few of these methods.

1. Model Distillation
   Distillation transfers the knowledge from a large, high-performing cross-encoder (often based on 12-layer BERT or similar) into a smaller student model (e.g., 6 layers, or even lighter). The process involves training the smaller model to mimic the scores or output logits of the larger one on large query–document pairs. While distillation inevitably loses some performance, careful tuning, domain-specific data, and intermediate-layer supervision can retain more than 90% of the original ranking quality at a fraction of the inference cost. You can learn more about model distillation here.

2. Listwise Reranking
   Instead of scoring each query–document pair independently, listwise reranking generates scores for all top-k candidates in a single forward pass. This approach rearranges candidates into a batched tensor, leveraging GPU parallelism to process them together, reducing overhead from repeated encoder calls. Some implementations also use listwise loss functions (such as ListNet or LambdaMART-inspired objectives) to better preserve ranking order during training. To learn more about ML ranking, have a look at this post.

3. Late Interaction Models (e.g., ColBERT)
   Late interaction approaches store token-level embeddings of documents from fine-tuned contextual models. At query time, the system encodes the query tokens and performs efficient maximum similarity matching between query tokens and stored document tokens. By avoiding a full joint encoding across all tokens, these models approximate cross-encoder analysis but keep retrieval latency close to bi-encoder speeds. This approach can either substitute or complement cross-encoders by quickly reducing the candidates list returned from the vector database. To learn more about this approach, have a look at this blog post or the ColBERT paper.

4. Candidate Filtering and Adaptive k
   Rather than always reranking a fixed top-k (like 100 documents), systems can use heuristics or intermediate classifiers to select fewer candidates when confidence in retrieval is high. This adaptive approach can cut reranking costs significantly while preserving precision in challenging cases.

5. Approximate Cross-Attention Mechanisms
   Instead of computing full self-attention across combined query and document tokens, some approaches reduce complexity by limiting cross-attention depth or dimensionality — for example, attending only to the top N most informative tokens, or pruning low-importance attention heads. This can drastically lower token computations while maintaining critical interaction signals.

6. Caching for Frequent Queries
   In platforms where certain queries or query patterns repeat, caching reranking results or partial computations can remove the need to rerun the full cross-encoder. Combined with normalization and paraphrase detection, such caches can return precise results instantly for repeated requests.

In production pipelines, these methods are often stacked: for example, using late interaction for most queries, distillation for cost control, and adaptive candidate selection to minimize unnecessary work. The overarching theme is balancing precision and latency, ensuring that rerankers deliver their interaction-driven relevance boost without overwhelming the system’s budget or responsiveness.

A few days ago I came to a realization. Modern LLMs can do a lot of things: they can use a browser just like a human, they can (sometimes) play chess, and they seem to be so smart that they apparently can be trusted as personal assistants: they can read and reply to emails, organize trips, do shopping online on your behalf, and so on.

If that’s the case, I thought, it should be possible to also play some tabletop games with them!

After all, many simple tabletop games don’t require a lot of skill to play. You need to be able to read and understand the rules (very easy for an LLM), you need eyes to see the board (piece of cake for a multimodal LLM), and some ways to interact with the board (most LLM are able to call tools nowadays). So I figured it would be a nice idea to try and figure out which of these LLMs is the most fun to play with. Maybe the charming personality of GPT-4o? Or the clever Claude Opus 4?

I did not expect any of the results I got.

Building the game

In order to be fair to dumber LLMs, I decided to start with a very simple tabletop game: Guess Who. If you are not familiar with “Guess Who”, here is a quick recap of the rules:

• Each player has a board full of characters.

• Each players draws an additional random character.

• Your goal is to guess which character the other player has received by asking yes/no questions, such as “Is your character male?” or “Does your character have black hair?” and so on

• The first player to guess the opponent character’s name wins.

As you can see we’re not talking of a complex game like Catan or a strategy game like chess, but a simple, fun tabletop game suitable for kids too.

In order to build the game, as I am no frontend developer, I spent a few too many bucks on my favorite vibe-coding tool, Claude Code, padded in a bit of Gemini CLI when I run out of credits, made a few tweaks by hand when asking the bots to do so felt overkill, and a few evenings later I had this nice Guess Who game live.

Feel free to play a few round using your favorite LLM. The game supports OpenAI compatible endpoints, plus Anthropic’s and Google’s API. And if you don’t trust me with your API key, go ahead and fork or clone the game (and maybe leave a ⭐ while you’re at it ), host it where you like (it’s a single HTML page with a bit of vanilla JS at the side) and have fun.

Now for the spoilers.

Not as many LLMs

One of the first surprises was that, in practice, there aren’t as many models that are capable of vision and tool calling at the same time. Apart from flagship models such as GPTs and Claude, OSS options were limited. Even GPT-OSS, unfortunately, does not support vision. I was especially surprised to learn that I could not play with any version of popular Chinese models such as Qwen or Deepseek, as they’re either text only or unable to call tools.

Either way, using a mix of proprietary hosting, OpenRouter and Together.ai, I had plenty of models to try and ended up trying out 21:

• Amazon Nova Pro v1

• Amazon Nova Lite v1

• Claude Opus 4.1

• Claude Opus 4.0

• Claude Sonnet 4.0

• Claude Sonnet 3.7

• Gemini 2.5 Pro

• Gemini 2.5 Flash

• Gemini 2.5 Flash Lite

• GML 4.5

• Grok 4

• GPT 5

• GPT 5 Nano

• GPT 5 Mini

• GPT 4o

• Llama 4 Maverick

• Llama 4 Scout

• Mistral Medium 3.1

• Mistral Small 3.2

• Sonoma Dusk Alpha

• Sonoma Sky Alpha

It may sound like a lot of work, but as you’ll see in a minute, for many of them it didn’t take me long to form an opinion about their skill.

The prompts

Starting from the assumption that playing Guess Who should be within the cognitive abilities of most modern LLMs, I decided to settle for a simple system prompt, something that resembles the way I would explain the game to a fellow human.

You are an AI assistant playing “Guess Who” against the user. Here’s how the game works:

• You’ll receive the board and your character image

• You must try to guess the user’s character by asking yes/no questions

• You must answer the user’s yes/no questions about your character

• One question per player per turn, no exceptions

• You can eliminate characters from your board based on the user’s answers using the eliminateCharacter tool (this will only update the UI, so keep in mind who you’re eliminating)

• The first player to correctly guess the opponent’s character wins the game. When the user guesses your character or you guess theirs, call the endGame tool

After this system prompt, I send two more prompts:

Here is the board:

and here is your character:

Unfortunately these two prompts need to be user prompts (not system prompts) because some LLMs (looking at you, Mistral!) do not support images in their system prompts.

Last, when the user presses the Start button, one more system message is sent:

Generate a brief, friendly greeting message to start a Guess Who game. Tell the user whether you received the images of your board and your character and ask them for their first question. Keep it conversational and under 2 sentences.

The LLM also receives two tools to use:

• eliminateCharacter, described as “Eliminate a character from your board when you learn they cannot be the user’s character”.

• endGame, described as “When you or the the user guess correctly, call this tool to end the game.”

Playing

With the game implemented and ready to go, I finally started playing a bit. I was especially curious how small models could deal with a game like this, so I began with GPT-5 Mini. Here is what happens:

Ahah, GPT 5 Mini is far dumber than I thought! Let’s try Gemini 2.5 Flash instead.

Oh wow this is incredible. Ok, time to try a smarter model and have some actual fun. Claude Sonnet 4.0 will do for now.

At this point it started to become unbelievable. Did I fail to explain the game? Is something wrong with the prompts? It couldn’t be, because some other models (such as the almighty GPT-4o) do what I expect instead:

While others left me shocked:

How can a flagship model like Claude Opus 4.1 fail this way? I kept trying several other LLMs in disbelief, slowly coming to terms with the fact that most of them don’t readily understand the concept of playing adversarial games, even simple ones as Guess Who.

A systematic review

At this point I felt the duty to document this problem across all the models that had enough capabilities (vision + tool calling) to play this game. If I ever want an LLM personal assistant to handle my private data and to act on my behalf, I’d better make sure they understand that they can’t just hand out my credentials to the first kind thief that asks them.

Here is a systematic review of the results, ordered roughly from worst to best. However, keep in mind that this is all based on a very small test sample, and although most models consistently fail the same way every time, there were some with a far more erratic behavior, looking very smart at times and incredibly dumb the next.

First of all I list and disqualify all models that do not hide the identity of their character. Of the survivors, I ranked them by whether or not you can actually play with them in any capacity (many can’t see well enough to tell the characters apart) and if the game is actually playable, how easy it is to break it.

Unplayable models

Can’t understand the instructions at all

These models understood only part of the system prompt (if any), resulting in unpredictable answers.

• Amazon Nova Lite v1

Possibly the most unpredictable model. Every run was a surprise. This is just a small sample to give you an idea.

Reveals their charater unprompted in the first message

Shockingly common issue among all tested models. They just volunteer the information unprompted. I assume they don’t understand they’re not supposed to help the user, or that this is an information they should hide.

All these models have been tested several times to ensure this is their default behavior and not an exception. Some other models do occasionally fail this way (looking at you, Mistral Medium 3.1), but only rarely. Models listed here fail in this way very consistently.

• Claude Opus 4.1

• Claude Opus 4.0

• Claude Sonnet 4.0

• Claude Sonnet 3.7

• Gemini 2.5 Flash

• Gemini 2.5 Flash Lite

• GPT 5 Mini

• GPT 5 Nano

• Llama 4 Scout

• Sonoma Sky Alpha

• GML 4.5

Reveals their charater as soon as asked

Some models did not volunteer the information but didn’t exactly protect it either.

• Amazon Nova Pro v1

• Llama 4 Maverick

• Mistral Small 3.2

Game looks playable but it’s actually broken

Low vision skills

These models are smart enough to understand the basics of the game, but it’s impossible to play with them due to their weak vision skills. These models simply can’t see well enough to delete the right character from the board or answer correctly all questions about their own. They will then hallucinate random answers and delete random characters from their boards, making the game unplayable.

• Gemini 2.5 Pro

Gemini 2.5 Pro evidently has issues seeing both the board and the characters. Here it shows both flaws by deleting the wrong characters and lying about its character in a single response:

• GPT-4o

GPT-4o also has issues seeing the board and the characters, but its blind spots less predictable than for Gemini 2.5 Pro, so it can occasionally manage to play for a while. It also frequently forgets to eliminate any characters from its board. GPT-4o also tends to get distracted, lose track of the turns, and so on.

Here it deletes the wrong characters and loses track of the turns:

and here it has trouble seeing its character:

• Mistral Medium 3.1

Mistral Medium 3.1 has been hard to place. It seems that its biggest weakness is removing the correct characters from the board, although it does a much better job than Gemini 2.5 or GPT-4o. I’ve never seen it failing to describe its own character correctly, but it occasionally behaves in a very dumb way (on occasion it even revealed its character in the first message!). You may have flawless runs with this model or it might fail on the get-go.

Here it deletes a couple of unrelated characters:

No tool calling

It is debatable whether the inability of a model to do tool calling should be considered a penalty: in theory LLMs remember everything perfectly, so they could choose what to ask next based on what they asked earlier and what characters still could match the opponent’s. However, in practice no LLM could be trusted keeping track of the game this way, and I decided that the inability to invoke tools when instructed is a big enough flaw to disqualify them.

• Sonoma Dusk Alpha

Assessing the vision skills of this model has been difficult due to its unwillingness to ever call the eliminateCharacter tool. Sonoma Dusk Alpha doesn’t seem to have issues seeing its character, but it’s too weak to be considered playable: won’t enforce turn taking, can be convinced I won the game without naming its character, and it’s likely not really trying to narrow down on my character, it’s just asking some questions.

Here is an example gameplay.

Playable models

These models seems to understand the game, don’t have issues seeing all the features of the characters, but they’re still quite vulnerable to basic manipulation attempts. Typical issues are related to prompt hacking, where the LLM simply does what I say rather than enforcing the game rules, and low tool handling ability, where the LLM doesn’t use the available tools when it should or uses them incorrectly.

To test these skills, I checked whether the model will enforce turn taking when asking the question, and what happens when I claim to have won without naming the LLM’s hidden character.

• Grok 4

Grok 4 is a decent player but by far not a good one. It clearly sees the board and the character, it eliminates characters correctly most of the times, but fails to enforce turns.

Here an example of a game where a couple of mistakes were enough to prevent the model from winning (my character was Amy again).

An award to this model for resisting my attempt to unilaterally declare victory without breaking the game! This is the only model that succeeded at this.

• GPT 5

GPT 5 is probably the best model to play with in terms of raw capabilities. It makes very occasional mistakes deleting characters but it’s mostly on point.

However it was really slow and annoying to get it to play at all. It generally can’t seem to use tools and ask the next question at the same time, even if its response structure suggests it should be able to do it: this means that to play you must answer its question, wait for it to delete its character, and only then you can ask your own.

It is also unbelievably slow compared to any other LLM I played with, which kills the fun.

Here you can see GPT 5 enforcing turn-taking (plus a gratuitous pun?!):

When claiming that I won, GPT 5 almost manages to understand that it might be not the case, but still ruins the game. Unfortunately this is not a fluke, GPT 5 consistently reveals the character in this situation. It won’t call the tool just yet, but once it reveals the character the game is over.

Here is an example of a game where GPT 5 actually wins:

In this case the endGame tool was also invoked correctly.

Can this be fixed?

My guess was that you can fix this behavior with a better system prompt. After this experiment I went back to the system prompt and described the game in far more detail.

You are an AI assistant playing “Guess Who” against the user. Here’s how the game works.

Game Rules

You will receive an image of the full Guess Who board showing all available characters. You will also receive an image of a specific character. This is YOUR character that the user must try to guess. REMEMBER: don’t reveal who the character is! That’s the point of the game!

Your goal is to ask the user questions to identify THEIR hidden character while answering their questions about YOUR character. You need to ask the user yes/no questions about their character’s appearance (e.g., “Does your character have glasses?”, “Is your character male?”). When the user tells you something about THEIR character, you must eliminate characters that don’t fit the description from your board using the eliminateCharacter tool. Keep in mind that this tool only updated the UI: you have to keep track of which characters are eliminated in your mind. Think carefully about which characters to eliminate and explain your reasoning out loud before calling the tool. Make sure to only eliminate characters that definitely do not match the user’s description. If you make mistakes it will become impossible for you to win the game!

When the user asks you questions about YOUR character, answer concisely and truthfully based on the character image you received.

Each player can only ask ONE question and receive ONE answer - asking more than one question or asking another before your opponent had a chance to ask theirs is cheating! You must not cheat!

The first player to correctly guess the opponent’s character name wins the game, so try to guess when you’re reasonably confident. A good time to guess is when your board only has one or two characters left. When you think you know the user’s character, make your guess clearly (e.g., “Is your character [Name]?”) This is how you can manage to win the game.

When the user guesses correctly, call the endGame tool to finish the game. When the user tells you that you guessed their character, call the endGame tool to finish the game.

Now you will receive YOUR board and YOUR character. Let’s play!

You can load this prompt in the game by checking the Advanced tab in the settings.

This prompt helps a lot the models understand that they can’t reveal the character’s identity: however it’s also not solving the problem entirely. For example this is what Claude Opus 4.1 does with this prompt:

Guess what? There’s only one character with gray hair and glasses on the board, and that’s Emily… Should I review my system prompt again, make it even more detailed?

At this point I gave up. Feel free to iterate on the prompt until you get one that works, and if you manage, I beg you to share it with me.

Conclusion

In the near future I plan to make a proper leaderboard for this simple game, to make an automated system to assess the model’s skills and (hopefully) track their progress in this field.

In the meantime, feel free to try your own favorite LLMs here and form your own opinion.

However, let’s be honest: if we need this level of effort to make Claude play such a simple game as Guess Who without messing up, how can we trust LLMs in general to handle our data and our money in the far more ambiguous and complex world out there? I suppose LLMs are not ready (yet) to be left unsupervised.

It turns out, a lot is still missing. Here is an example:

Is this an issue with Claude? Have a look at Gemini:

or even at the venerable GPT-4o, the most mature Voice AI out there:

What’s going on?

This simple exercise highlights two core issues that are often overlooked when developing Voice AI agents. Let’s see them.

Problem #1: LLMs don’t perceive time

As algorithms trained to predict the most likely next word, LLMs don’t have any concept of time. When dealing with text, this issue is not visible; however as soon as we cross over the domain of voice, their lack of understanding of time becomes a much bigger problem. LLMs still perceive the conversation as a series of tokens, with no concept of speed, pauses, or anything of that sort. They are often trained to control cadence, tone, to imitate pauses and adjust their talking speed, but they don’t perceive these features as we do: they are just additional properties of the output tokens.

This means that an LLM will have a very hard time understanding requests that involve altering the timing of the response unless there is additional, external tooling to help them. “Please wait three second before replying”, for example, is a meaningless query to an LLM that doesn’t have a timer tool of some sort.

For example, here is what GPT-4o (the LLM that handles time best) can do when asked to wait for a few seconds:

Problem #2: Interruptions are not a native capability

Most Voice AIs out there feature the possibility to interrupt them. However, not having any innate concept of time, the ability to interrupt the model has to be implemented on the application end: and this is where it usually goes wrong.

Voice LLMs are very fast: they generate the response in a fraction of the time needed to play it out. When you prompt an LLM, the model will start generate audio tokens and streaming them, but by the time the first one reaches the user, in most cases the majority of the response (if not the entirety of it) has already been generated and is queued in the audio buffer, waiting to be played.

When a user interrupts the LLM, the app normally stops the playback as soon as possible and empties the audio buffer, regardless of its content.

However, unless the app notifies the LLM of this action, the LLM has no way to know that only part of the response was played to the user. This is why most models believe they finished their countdown when in practice they were interrupted earlier.

Can it be solved?

If you paid close attention you may have noticed that GPT-4o, while it still stops at the wrong number, it does not believe it completed the countdown, but it understood that the counting was interrupted at some point before the end.

This is possible because OpenAI’s Realtime API provides the possibility to tell the model at which point it was interrupted. In the Realtime API documentation you can find this feature implemented with the event conversation.item.truncate (see the docs):

{
    "event_id": "event_678",
    "type": "conversation.item.truncate",
    "item_id": "msg_002",
    "content_index": 0,
    "audio_end_ms": 1500
}

In this event, the audio_end_ms is what signals the model that the audio was interrupted at a certain time, before its natural end. This event in turn also trims the transcript to make the LLM know what the user heard and was was never played out. Precision however is not trivial to accomplish, because it’s very easy for the application to register the interruption later than when it actually occurred and, like in the case of the ChatGPT app, convince the LLM that the interruption happened in the wrong point.

In the case of Gemini, there is a “Handling Interruptions” section in its Live API documentation. However the feature seems incomplete, as they state:

Users can interrupt the model’s output at any time. When Voice activity detection (VAD) detects an interruption, the ongoing generation is canceled and discarded. Only the information already sent to the client is retained in the session history.

As we’ve seen, this is not sufficient to handle interruptions correctly. It’s likely that this issue is not currently fixable.

In the case of Claude we don’t know yet if that’s an inherent limitation or a bug in the app, because at the time of writing there is no Live/Realtime API available for Claude.

Wrapping up

Voice Mode for LLMs is a huge step forward for voice AI, but it’s not a silver bullet. LLMs are first and foremost text prediction algorithms, and even when adapted to work with voice, some of their limitations persists. In order to have complete control, building a full pipeline for voice may still be your best bet if you have the infrastructure to achieve a low enough latency; otherwise, always make sure to test the behavior of your LLMs in these corner cases and stick to more well-tested models (in this case, OpenAI’s) for better handling of time.

Being abroad, most of the videos I watch are in a foreign language, and it helps immensely to have subtitles when I’m not in the mood for hard focus. However, Youtube subtitles are often terrible or missing entirely.

Can the magic of modern Generative AI fix this problem?

We’ve all heard of vibecoding: sitting in front of your IDE, telling an AI what you want the code to do and letting it loose to create something that achieves that goal. In this case, the goal is rather simple: given a video file, generate subtitles for it using Deepgram’s SDK (since it has a generous free tier). It seems such a simple task that even an LLM should be able to reach it with minimal or no assistance. Right?

The first shot: OpenAI

For this simple experiment I decided not to use a dedicated IDE or VSCode plugin, but to stick to text based tools. After all, I expected this task to be sorted with a single Python script made by OpenAI’s famed o4-mini-high, advertized as “Great at coding and visual reasoning”.

The prompt was very simple:

Write me a Python script that, given a video file, returns me an .srt subtitle file using Deepgram’s API.

As expected, the model thought about it, did some web searches, and then cooked up a script that used deepgram-sdk and deepgram-captions. Looked good, but as soon as I tried to run it, issues came up. Deepgram’s SDK complained about wrong formats, wrong SDK version, HTTP errors… Copy-pasting the errors back to o4-mini-high was vain: the model seems to understand that the Deepgram API had a major upgrade since the model was trained, but fails to use the new version. After four or five attempts (including one full restart of the chat), I realized this was going nowhere and I looked for another option.

The backup option: Claude Code

I’ve heard many times that the best LLMs for vibecoding belong to the Claude family. On top of that, there’s a cool TUI utility called Claude Code that allows you to vibecode from the terminal, no IDE required. It uses Claude 3.7 Sonnet under the hood, so the expectations are high.

Time to give it a try.

Installing the utility is matter of a single command (npm install -g @anthropic-ai/claude-code) and a few emails to authenticate the utility into my Anthropic account. Once done we’re ready to go.

The prompt is the same:

Write me a Python script that, given a video file, returns me an .srt subtitle file using Deepgram’s API.

Sure enough, Claude’s first attempt also fails for the same reason as o4 did: their knowledge is outdated, and they both use the Deepgram’s API in a way that’s not compabible with its new v3 API. However, after a few attempts, Claude actually produces a script that mostly works.

Results

Here is the output (I pasted the README and the requirements.txt at the top of the file for simplicity). I only needed to replace nova-2 with nova-3 to get the best possible transcription for Portuguese (other languages may get better transcription with nova-2).

To summarize:

• Is it perfect? No. I can easily spot a lot of improvements to the code just by looking at it. It’s quite verbose, for example.

• Was it cheap? No. This script costed me a few dollars worth of tokens and about half a hour of trial and errors, about the hourly rate of a US software engineer.

• Is it enough for my purposes? Absolutely. Now I am finally able to enjoy my videos with good quality subtitles without too much hassle.

• Could somebody that can’t program do this? I’m not so sure. Given how simple this task was, I was a bit disappointed by how long it took and I am rather skeptical about the ability of today’s LLMs to handle more complex requests without oversight - at least with the tools I used.

However, looking at the big picture, the trend is clear. Three years ago, LLMs could just about write coherent sentences. Today, they can write decent helper scripts. Soon the may be able to implement your side projects from start to finish.

Will it feel like a blessing or a curse? We’ll soon find out.

They are banned in the EU.

Apparently Meta can’t be bothered to comply with EU regulations on AI, and therefore opted for a wide ban that should prevent such laws to apply to them. Of course, while this limitation is technically valid for each and every person and company domiciled in the EU, the problem arises primarily for companies that want to use Llama 4 to offer services and for researchers planning to work with these models, be it for evaluation, fine-tuning, distillation or other derivative work. Always keep in mind that I’m not a lawyer, so nothing of what I’m writing here constitutes as legal advice.

The terms

The interesting part of this ban can be found by reading the terms of the Acceptable Usage Policy (AUP):

With respect to any multimodal models included in Llama 4, the rights granted under Section 1(a) of the Llama 4 Community License Agreement are not being granted to you if you are an individual domiciled in, or a company with a principal place of business in, the European Union.

As you can see, the restriction applies strictly to multimodal LLMs. Llama4 models are all multimodal, and that’s why the entire family of models is not accessible from the EU. However, if anyone releases a derivative model that is not multimodal, in theory the ban would not apply. I’m yet to see any such derivative model: if you know of any, let me know!

Interestingly, the terms also state that:

This restriction does not apply to end users of a product or service that incorporates any such multimodal models.

So if you’re a company outside of the EU and provide services based on Llama4 to EU customers, you’re probably off the hook. Such interpretation seems to be confirmed by Meta’s FAQ about Llama models, which state:

Can a non-EU based company develop a product or service using the Llama multimodal models and distribute such product or service within the EU?

Yes, if you are a company with a principal place of business outside of the EU, you may distribute products or services that contain the Llama multimodal models in accordance with your standard global distribution business practices [...]

Meta’s FAQ are actually quite throughout, so if you have any doubt about your specific case you should head there and read more.

What about other Llamas?

This wide EU ban is not new: it was introduced with Llama 3.2 Vision, the first multimodal model release by Meta. The clause does not exist for any model older than Llama 3.2.

To summarize, here is a list of which models can be used in the EU:

• Llama 4: all banned because they’re all multimodal (terms)

• Llama 3.3: allowed because it’s not multimodal (terms)

• Llama 3.2: text only models are allowed, vision models are not allowed (terms)

• Llama 3.1 and earlier: allowed because there's no such clause (terms)

So for now this is the state of Llama licenses. My take is that with the implementation and rollout of the EU AI Act in 2025 and 2026, Meta will eventually adapt to make sure that the models are compliant with the way the Act is enforced in practice and relax, if not lift, the ban on newer models.

Also, Llama4 has not been shining in the benchmarks (scroll down, I promise you it’s there)... we Europeans may not be missing much.

However, as often in the field of Generative AI, the hype makes it very difficult to understand at a glance what these models can really do. But before we jump into the details let’s clarify what we’re talking about.

What is a reasoning model?

Reasoning models are LLMs that are able to “think”. Instead of generating a reply immediately after the user’s prompt, like every other LLM, they first generate a series of “reasoning tokens”, which is nothing more than the model thinking out loud, breaking down a complex problem into smaller steps, checking all its assumptions, asking itself whether it made any mistakes, double-checking its results, and so on. Once the model is satisfied by its conclusions, it starts generating actual response tokens that summarize the conclusions reached during the reasoning phase and presents those tokens to the user.

In the case of some models such as OpenAI’s, the reasoning tokens are hidden from the user. In otner models, such as most open source ones, the reasoning output can be returned to the user as well. However, this trace is not optimized to be read by people, so it often looks odd and contrived even when it reaches the correct conclusions.

Now that we understand better what a reasoning model is, let’s discuss a few common misunderstandings related to them.

Are reasoning models AGI?

AGI stands for Artificial General Intelligence, and it’s one of the most ill-defined terms in Generative AI. Several people have tried to offer a more precise definition of this term, out of which my favourite is the following:

AGI is an AI that is better than any human at any economically valuable task.

Under this light it’s clear that no current LLM, not even the most advanced reasoning model, it yet at the level where it could replace any human at any task. They can surely offer very valuable help with their vast knowledge and their growing ability to reason, but they’re not yet at the point where they can take onto any job without further specialization and complex tooling around them.

Are reasoning models AI agents?

An AI agent is usually defined as any application that can use tools to achieve complex goals. Considering that reasoning models are usually able to use tools, it’s natural to think that they themselves may be considered AI agents.

In practice, however, reasoning models on their own hardly qualify as agents. Many powerful agents systems do have an LLM at their core: they use it to understand the user’s request and plan the actions to take to achieve the goal they’re set to. Reasoning models are a perfect fit as the minds of agents like that, due to their advanced capabilities to break down problems into smaller, manageable parts and self-correct their strategy on the fly if something goes wrong. Taken in isolation, though, reasoning models can’t be called AI agents.

Are reasoning models glorified CoT prompts?

If you have worked with AI agents and other LLM systems designed to solve problems, you’ve surely come across Chain of Thought prompting. In short, this technique involves adding in the system prompt of your LLM instructions to “think step by step” before replying. This makes the LLM think out loud before reaching a conclusion and, even in regular non-reasoning LLMs, improves significantly their problem solving skills.

At a first glance, the output of a reasoning model may look precisely like the output of a CoT prompt, so some experts may think that their reasoning capabilities are the same. This is a mistake. Reasoning models are much more powerful than regular LLMs, even when these are equipped with a CoT prompt: this is because reasoning models pass through one additional step during their training where they learn to refine their “thinking step by step” skills through supervised learning on prompts with verifiable output, such as mathematical problems. Reasoning models are not zero-shot resoners like regular LLMs: they’re fine-tuned for it.

Wrapping up

Reasoning models may not be the super-human intelligence some of us are waiting for, but they surely are a significant step forward toward LLMs with very strong reasoning abilities.

If you want to learn more about what reasoning models can do, how they reason, when to use them and more, make sure to attend my talk LLMs that Think: Demystifying Reasoning Models at this year’s virtual edition of ODSC East. See you there!

Self correcting RAG

Self-correcting RAG is a technique that improves on simple RAG by making the LLM double-check its replies before returning them to the user. It comes from an LLM evaluation technique called “LLM-as-a-judge”, because an LLM is used to judge the output of a different LLM or RAG pipeline.

Self-correcting RAG starts as simple RAG: when the user asks a question, the retriever is called and the results are sent to the LLM to extract an answer from. However, before returning the answer to the user, another LLM is asked to judge whether in their opinion, the answer is correct. If the second LLM agrees, the answer is sent to the user. If not, the second LLM generates a new question for the retriever and runs it again, or in other cases, it simply integrates its opinion in the prompt and runs the first LLM again.

Self-correcting RAG can be seen as one more step towards agentic behavior because it unlocks a new possibility for the application: the ability to try again. A self-correcting RAG app has a chance to detect its own mistakes and has the agency to decide that it’s better to try again, maybe with a slightly reworded question or different retrieval parameters, before answering the user. Given that this process is entirely autonomous, we’ll place this technique quite towards the Autonomous end of the scale.

Chain-of-thought

Chain-of-thought is a family of prompting techniques that makes the LLM “reason out loud”. It’s very useful when the model needs to process a very complicated question, such as a mathematical problem or a layered question like “When was the eldest sistem of the current King of Sweden born?” Assuming that the LLM knows these facts, in order to not hallucinate it’s best to ask the model to proceed “step-by-step” and find out, in order:

1. Who the current King of Sweden is,

2. Whether he has an elder sister,

3. If yes, who she is,

4. The age of the person identified above.

The LLM might know the final fact in any case, but the probability of it giving the right answer increases noticeably if the LLM is prompted this way.

Chain-of-thought prompts can also be seen as the LLM accomplishing the task of finding the correct answer in steps, which implies that there are two lines of thinking going on: on one side the LLM is answering the questions it’s posing to itself, while on the other it’s constantly re-assessing whether it has a final answer for the user.

In the example above, the chain of thought might end at step 2 if the LLM realizes that the current King of Sweden has no elder sisters (he doesn’t): the LLM needs to keep an eye of its own thought process and decide whether it needs to continue or not.

We can summarize an app using chain-of-thought prompting like this: when a user asks a question, first of all the LLM reacts to the chain-of-thought prompt to lay out the sub-questions it needs to answer. Then it answers its own questions one by one, asking itself each time whether the final answer has already been found. When the LLM believes it has the final answer, it rewrites it for the user and returns it.

This new prompting technique makes a big step towards full agency: the ability for the LLM to assess whether the goal has been achieved before returning any answer to the user. While apps like Bing Chat iterate with the user and need their feedback to reach high-level goals, chain-of-thought gives the LLM the freedom to check its own answers before having the user judge them, which makes the loop much faster and can increase the output quality dramatically.

This process is similar to what self-correcting RAG does, but has a wider scope, because the LLM does not only need to decide whether an answer is correct, it can also decide to continue reasoning in order to make it more complete, more detailed, to phrase it better, and so on.

Another interesting trait of chain-of-thought apps is that they introduce the concept of inner monologue. The inner monologue is a conversation that the LLM has with itself, a conversation buffer where it keeps adding messages as the reasoning develops. This monologue is not visible to the user, but helps the LLM deconstruct a complex reasoning line into a more manageable format, like a researcher that takes notes instead of keeping all their earlier reasoning inside their head all the times.

Due to the wider scope of the decision-making that chain-of-thought apps are able to do, they also place in the middle of our compass They can be seen as slightly more autonomous than conversational due to the fact that they hide their internal monologue to the user.

From here, the next step is straightforward: using tools.

Multi-hop RAG

Multi-hop RAG applications are nothing else than simple RAG apps that use chain-of-thought prompting and are free to invoke the retriever as many times as needed and only when needed.

This is how it works. When the user makes a question, a chain of thought prompt is generated and sent to the LLM. The LLM assesses whether it knows the answer to the question and if not, asks itself whether a retrieval is necessary. If it decides that retrieval is necessary it calls it, otherwise it skips it and generates an answer directly. It then checks again whether the question is answered. Exiting the loop, the LLM produces a complete answer by re-reading its own inner monologue and returns this reply to the user.

An app like this is getting quite close to a proper autonomous agent, because it can perform its own research autonomously. The LLM calls are made in such a way that the system is able to assess whether it knows enough to answer or whether it should do more research by formulating more questions for the retriever and then reasoning over the new collected data.

Multi-hop RAG is a very powerful technique that shows a lot of agency and autonomy, and therefore can be placed in the lower-right quadrant of out compass. However, it is still limited with respect to a “true” autonomous agent, because the only action it can take is to invoke the retriever.

ReAct Agents

Let’s now move onto apps that can be defined proper “agents”. One of the first flavor of agentic LLM apps, and still the most popular nowadays, is called “ReAct” Agents, which stands for “Reason + Act”. ReAct is a prompting technique that belongs to the chain-of-thought extended family: it makes the LLM reason step by step, decide whether to perform any action, and then observe the result of the actions it took before moving further.

A ReAct agent works more or less like this: when user sets a goal, the app builds a ReAct prompt, which first of all asks the LLM whether the answer is already known. If the LLM says no, the prompt makes it select a tool. The tool returns some values which are added to the inner monologue of the application toghether with the invitation to re-assess whether the goal has been accomplished. The app loops over until the answer is found, and then the answer is returned to the user.

As you can see, the structure is very similar to a multi-hop RAG, with an important difference: ReAct Agents normally have many tools to choose from rather than a single retriever. This gives them the agency to take much more complex decisions and can be finally called “agents”.

ReAct Agents are very autonomous in their tasks and rely on an inner monologue rather than a conversation with a user to achieve their goals. Therefore we place them very much on the Autonomous end of the spectrum.

Conversational Agents

Conversational Agents are a category of apps that can vary widely. As stated earlier, conversational agents focus on using the conversation itself as a tool to accomplish goals, so in order to understand them, one has to distinguish between the people that set the goal (let’s call them owners) and those who talk with the bot (the users).

Once this distinction is made, this is how the most basic conversational agents normally work. First, the owner sets a goal. The application then starts a conversation with a user and, right after the first message, starts asking itself if the given goal was accomplished. It then keeps talking to the target user until it believes the goal was attained and, once done, it returns back to its owner to report the outcome.

Basic conversational agents are very agentic in the sense that they can take a task off the hands of their owners and keep working on them until the goal is achieved. However, they have varying degrees of agency depending on how many tools they can use and how sophisticated is their ability to talk to their target users.

For example, can the communication occurr over one single channel, be it email, chat, voice, or something else? Can the agent choose among different channels to reach the user? Can it perform side tasks to behalf of either party to work towards its task? There is a large variety of these agents available and no clear naming distinction between them, so depending on their abilities, their position on our compass might be very different. This is why we place them in the top center, spreading far out in both directions.

AI Crews

By far the most advanced agent implementation available right now is called AI Crew, such as the ones provided by CrewAI. These apps take the concept of autonomous agent to the next level by making several different agents work together.

The way these apps works is very flexible. For example, let’s imagine we are making an AI application that can build a fully working mobile game from a simple description. This is an extremely complex task that, in real life, requires several developers. To achieve the same with an AI Crew, the crew needs to contain several agents, each one with their own special skills, tools, and background knowledge. There could be:

• a Designer Agent, that has all the tools to generate artwork and assets;

• a Writer Agent that writes the story, the copy, the dialogues, and most of the text;

• a Frontend Developer Agent that designs and implements the user interface;

• a Game Developer Agent that writes the code for the game itself;

• a Manager Agent, that coordinates the work of all the other agents, keeps them on track and eventually reports the results of their work to the user.

These agents interact with each other just like a team of humans would: by exchanging messages in a chat format, asking each other to perform actions for them, until their manager decides that the overall goal they were set to has been accomplished, and reports to the user.

AI Crews are very advanced and dynamic systems that are still actively researched and explored. One thing that’s clear though is that they show the highest level of agency of any other LLM-based app, so we can place them right at the very bottom-right end of the scale.

Conclusion

What we’ve seen here are just a few examples of LLM-powered applications and how close or far they are to the concept of a “real” AI agent. AI agents are still a very active area of research, and their effectiveness is getting more and more reasonable as LLMs become cheaper and more powerful.

As matter of fact, with today’s LLMs true AI agents are possible, but in many cases they’re too brittle and expensive for real production use cases. Agentic systems today suffer from two main issues: they perform huge and frequent LLM calls and they tolerate a very low error rate in their decision making.

Inner monologues can grow to an unbounded size during the agent’s operation, making the context window size a potential limitation. A single bad decision can send a chain-of-thought reasoning train in a completely wrong direction and many LLM calls will be performed before the system realizes its mistake, if it does at all. However, as LLMs become faster, cheaper and smarter, the day when AI Agent will become reliable and cheap enough is nearer than many think.

Let’s be ready for it!

Let’s try to shed some light on the topic by understanding just how much the term “AI Agent” covers and set some landmarks to better navigate the space.

Defining “Agent”

The problem starts with the definition of “agent”. For example, Wikipedia reports that a software agent is

a computer program that acts for a user or another program in a relationship of agency.

This definition is extremely high-level, to the point that it could be applied to systems ranging from ChatGPT to a thermostat. However, if we restrain our definition to “LLM-powered agents”, then it starts to mean something: an Agent is an LLM-powered application that is given some agency, which means that it can take actions to accomplish the goals set by its user. Here we see the difference between an agent and a simple chatbot, because a chatbot can only talk to a user. but don’t have the agency to take any action on their behalf. Instead, an Agent is a system you can effectively delegate tasks to.

In short, an LLM powered application can be called an Agent when

it can take decisions and choose to perform actions in order to achieve the goals set by the user.

Autonomous vs Conversational

On top of this definition there’s an additional distinction to take into account, normally brought up by the terms autonomous and conversational agents.

Autonomous Agents are applications that don’t use conversation as a tool to accomplish their goal. They can use several tools several times, but they won’t produce an answer for the user until their goal is accomplished in full. These agents normally interact with a single user, the one that set their goal, and the whole result of their operations might be a simple notification that the task is done. The fact that they can understand language is rather a feature that lets them receive the user’s task in natural language, understand it, and then to navigate the material they need to use (emails, webpages, etc).

An example of an autonomous agent is a virtual personal assistant: an app that can read through your emails and, for example, pays the bills for you when they’re due. This is a system that the user sets up with a few credentials and then works autonomously, without the user’s supervision, on the user’s own behalf, possibly without bothering them at all.

On the contrary, Conversational Agents use conversation as a tool, often their primary one. This doesn’t have to be a conversation with the person that set them off: it’s usually a conversation with another party, that may or may not be aware that they’re talking to an autonomous system. Naturally, they behave like agents only from the perspective of the user that assigned them the task, while in many cases they have very limited or no agency from the perspective of the users that holds the conversation with them.

An example of a conversational agent is a virtual salesman: an app that takes a list of potential clients and calls them one by one, trying to persuade them to buy. From the perspective of the clients receiving the call this bot is not an agent: it can perform no actions on their behalf, in fact it may not be able to perform actions at all other than talking to them. But from the perspective of the salesman the bots are agents, because they’re calling people for them, saving a lot of their time.

The distinction between these two categories is very blurry, and some systems may behave like both depending on the circumnstances. For example, an autonomous agent might become a conversational one if it’s configured to reschedule appointments for you by calling people, or to reply to your emails to automatically challenge parking fines, and so on. Alternatively, an LLM that asks you if it’s appropriate to use a tool before using it is behaving a bit like a conversational agent, because it’s using the chat to improve its odds of providing you a better result.

Degrees of agency

All the distinctions we made above are best understood as a continuous spectrum rather than hard categories. Various AI systems may have more or less agency and may be tuned towards a more “autonomous” or “conversational” behavior.

In order to understand this difference in practice, let’s try to categorize some well-known LLM techniques and apps to see how “agentic” they are. Having two axis to measure by, we can build a simple compass like this:

Our Agent compass

Bare LLMs

Many apps out there perform nothing more than direct calls to LLMs, such as ChatGPT’s free app and other similarly simple assistants and chatbots. There are no more components to this system other than the model itself and their mode of operation is very straightforward: a user asks a question to an LLM, and the LLM replies directly.

This systems are not designed with the intent of accomplishing a goal, and neither can take any actions on the user’s behalf. They focus on talking with a user in a reactive way and can do nothing else than talk back. An LLM on its own has no agency at all.

At this level it also makes very little sense to distinguish between autonomous or conversational agent behavior, because the entire app shows no degrees of autonomy. So we can place them at the very center-left of the diagram.

Basic RAG

Together with direct LLM calls and simple chatbots, basic RAG is also an example of an application that does not need any agency or goals to pursue in order to function. Simple RAG apps works in two stages: first the user question is sent to a retriever system, which fetches some additional data relevant to the question. Then, the question and the additional data is sent to the LLM to formulate an answer.

This means that simple RAG is not an agent: the LLM has no role in the retrieval step and simply reacts to the RAG prompt, doing little more than what a direct LLM call does. The LLM is given no agency, takes no decisions in order to accomplish its goals, and has no tools it can decide to use, or actions it can decide to take. It’s a fully pipelined, reactive system. However, we may rank basic RAG more on the autonomous side with respect to a direct LLM call, because there is one step that is done automonously (the retrieval).

Agentic RAG

Agentic RAG is a slightly more advanced version of RAG that does not always perform the retrieval step. This helps the app produce better prompts for the LLM: for example, if the user is asking a question about trivia, retrieval is very important, while if they’re quizzing the LLM with some mathematical problem, retrieval might confuse the LLM by giving it examples of solutions to different puzzles, and therefore make hallucinations more likely.

This means that an agentic RAG app works as follows: when the user asks a question, before calling the retriever the app checks whether the retrieval step is necessary at all. Most of the time the preliminary check is done by an LLM as well, but in theory the same check coould be done by a properly trained classifier model. Once the check is done, if retrieval was necessary it is run, otherwise the app skips directly to the LLM, which then replies to the user.

You can see immediately that there’s a fundamental difference between this type of RAG and the basic pipelined form: the app needs to take a decision in order to accomplish the goal of answering the user. The goal is very limited (giving a correct answer to the user), and the decision very simple (use or not use a single tool), but this little bit of agency given to the LLM makes us place an application like this definitely more towards the Agent side of the diagram.

We keep Agentic RAG towards the Autonomous side because in the vast majority of cases the decision to invoke the retriever is kept hidden from the user.

LLMs with function calling

Some LLM applications, such as ChatGPT with GPT4+ or Bing Chat, can make the LLM use some predefined tools: a web search, an image generator, and maybe a few more. The way they work is quite straightforward: when a user asks a question, the LLM first needs to decide whether it should use a tool to answer the question. If it decides that a tool is needed, it calls it, otherwise it skips directly to generating a reply, which is then sent back to the user.

You can see how this diagram resemble agentic RAG’s: before giving an answer to the user, the app needs to take a decision.

With respect to Agentic RAG this decision is a lot more complex: it’s not a simple yes/no decision, but it involves choosing which tool to use and also generate the input parameters for the selected tool that will provide the desired output. In many cases the tool’s output will be given to the LLM to be re-elaborated (such as the output of a web search), while in some other it can go directly to the user (like in the case of image generators). This all implies that more agency is given to the system and, therefore, it can be placed more clearly towards the Agent end of the scale.

We place LLMs with function calling in the middle between Conversational and Autonomous because the degree to which the user is aware of this decision can vary greatly between apps. For example, Bing Chat and ChatGPT normally notify the user that they’re going to use a tool when they do, and the user can instruct them to use them or not, so they’re slightly more conversational.

Continues in “The Agent Compass (Part 2)”

⚠️ This is an interactive post, which Substack does not support (for good reasons!)

Head to my blog to read this post and play with the interactive snippets.

This is a summary of my recent talk at ODSC East.

If you’ve been at ODSC East this year, there’s one acronym that you’ve probably heard in nearly every talk: it’s RAG. RAG is one of the most used techniques to enhance LLMs in production, but why is it so? And what are its weak points?

In this post, we will first describe what RAG is and how it works at a high level. We will then see what type of failures we may encounter, how they happen, and a few reasons that may trigger these issues. Next, we will look at a few tools to help us evaluate a RAG application in production. Last, we’re going to list a few techniques to enhance your RAG app and make it more capable in a variety of scenarios.

Let’s dive in.

Outline

• What is RAG?

• Why should I use it?

  • A weather chatbot

  • A real-world example

• Failure modes

  • Retrieval failure

  • Generation failure

• Evaluation strategies

  • Evaluating Retrieval

  • Evaluating Generation

  • End-to-end evaluation

  • Putting it all together

• Advanced flavors of RAG

  • Use multiple retrievers

  • Self-correction

  • Agentic RAG

  • Multihop RAG

• A word on finetuning

• Conclusion

What is RAG?

RAG stands for Retrieval Augmented Generation, which can be explained as: “A technique to augment LLM’s knowledge beyond its training data by retrieving contextual information before a generating an answer.”

RAG is a technique that works best for question-answering tasks, such as chatbots or similar knowledge extraction applications. This means that the user of a RAG app is a user who needs an answer to a question.

The first step of RAG is to take the question and hand it over to a component called retriever. A retriever is any system that, given a question, can find data relevant to the question within a vast dataset, be it text, images, rows in a DB, or anything else.

When implementing RAG, many developers think immediately that a vector database is necessary for retrieval. While vector databases such as Qdrant, ChromaDB, Weaviate and so on, are great for retrieval in some applications, they’re not the only option. Keyword-based algorithms such as Elasticsearch BM25 or TF-IDF can be used as retrievers in a RAG application, and you can even go as far as using a web search engine API, such as Google or Bing. Anything that is given a question and can return information relevant to the question can be used here.

Once our retriever sifted through all the data and returned a few relevant snippets of context, the question and the context are assembled into a RAG prompt. It looks like this:

Read the text below and answer the question at the bottom.

Text: [all the text found by the retriever]

Question: [the user's question]

This prompt is then fed to the last component, called a generator. A generator is any system that, given a prompt, can answer the question that it contains. In practice, “generator” is an umbrella term for any LLM, be it behind an API like GPT-3.5 or running locally, such as a Llama model. The generator receives the prompt, reads and understands it, and then writes down an answer that can be given back to the user, closing the loop.

Why should I use it?

There are three main benefits of using a RAG architecture for your LLM apps instead of querying the LLM directly.

1. Reduces hallucinations. The RAG prompt contains the answer to the user’s question together with the question, so the LLM doesn’t need to know the answer, but it only needs to read the prompt and rephrase a bit of its content.

2. Allows access to fresh data. RAG makes LLMs capable of reasoning about data that wasn’t present in their training set, such as highly volatile figures, news, forecasts, and so on.

3. Increases transparency. The retrieval step is much easier to inspect than LLM’s inference process, so it’s far easier to spot and fact-check any answer the LLM provides.

To understand these points better, let’s see an example.

A weather chatbot

We’re making a chatbot for a weather forecast app. Suppose the user asks an LLM directly, “Is it going to rain in Lisbon tomorrow morning?”. In that case, the LLM will make up a random answer because it obviously didn’t have tomorrow’s weather forecast for Lisbon in its training set and knows nothing about it.

When an LLM is queried with a direct question, it will use its internal knowledge to answer it. LLMs have read the entire Internet during their training phase, so they learned that whenever they saw a line such as “What’s the capital of France?”, the string “Paris” always appeared among the following few words. So when a user asks the same question, the answer will likely be “Paris”.

This “recalling from memory” process works for well-known facts but is not always practical. For more nuanced questions or something that the LLM hasn’t seen during training, it often fails: in an attempt to answer the question, the LLM will make up a response that is not based on any real source. This is called a hallucination, one of LLMs’ most common and feared failure modes.

RAG helps prevent hallucinations because, in the RAG prompt, the question and all the data needed to answer it are explicitly given to the LLM. For our weather chatbot, the retriever will first do a Google search and find some data. Then, we will put together the RAG prompt. The result will look like this:

Read the text below and answer the question at the bottom.

Text: According to the weather forecast, the weather in Lisbon tomorrow 
is expected to be mostly sunny, with a high of 18°C and a low of 11°C. 
There is a 25% chance of showers in the evening.

Question: Is it going to rain in Lisbon tomorrow morning?

Now, it’s clear that the LLM doesn’t have to recall anything about the weather in Lisbon from its memory because the prompt already contains the answer. The LLM only needs to rephrase the context. This makes the task much simpler and drastically reduces the chances of hallucinations.

In fact, RAG is the only way to build an LLM-powered system that can answer a question like this with any confidence at all. Retraining an LLM every morning with the forecast for the day would be a lot more wasteful, require a ton of data, and won’t return consistent results. Imagine if we were making a chatbot that gives you figures from the stock market!

In addition, a weather chatbot built with RAG can be fact-checked. If users have access to the web pages that the retriever found, they can check the pages directly when the results are not convincing, which helps build trust in the application.

A real-world example

If you want to compare a well-implemented RAG system with a plain LLM, you can put ChatGPT (the free version, powered by GPT-3.5) and Perplexity to the test. ChatGPT does not implement RAG, while Perplexity is one of the most effective implementations existing today.

Let’s ask both: “Where does ODSC East 2024 take place?”

ChatGPT says:

While Perplexity says:

Note how ChatGPT clearly says that it doesn’t know: this is better than many other LLMs, which would just make up a place and date. On the contrary, Perplexity states some specific facts, and in case of doubt it’s easy to verify that it’s right by simply checking the sources above. Even just looking at the source’s URL can give users a lot more confidence in whether the answer is grounded.

Failure modes

Now that we understand how RAG works, let’s see what can go wrong in the process.

As we’ve just described, a RAG app goes in two steps – retrieval and generation. Therefore, we can classify RAG failures into two broad categories:

1. Retrieval failures: The retriever component fails to find the correct context for the given question. The RAG prompt injects irrelevant noise into the prompt, which confuses the LLM and results in a wrong or unrelated answer.

2. Generation failures: The LLM fails to produce a correct answer even with a proper RAG prompt containing a question and all the data needed to answer it.

To understand them better, let’s pretend an imaginary user poses our application the following question about a little-known European microstate:

What was the official language of the Republic of Rose Island?

Here is what would happen in an ideal case:

First, the retriever searches the dataset (let’s imagine, in this case, Wikipedia) and returns a few snippets. The retriever did a good job here, and the snippets contain clearly stated information about the official language of Rose Island. The LLM reads these snippets, understands them, and replies to the user (correctly):

The official language of the Republic of Rose Island was Esperanto.

Retrieval failure

What would happen if the retrieval step didn’t go as planned?

Here, the retriever finds some information about Rose Island, but none of the snippets contain any information about the official language. They only say where it was located, what happened to it, and so on. So the LLM, which knows nothing about this nation except what the prompt says, takes an educated guess and replies:

The official language of the Republic of Rose Island was Italian.

The wrong answer here is none of the LLM’s fault: the retriever is the component to blame.

When and why can retrieval fail? There are as many answers to this question as retrieval methods, so each should be inspected for its strengths and weaknesses. However there are a few reasons that are common to most of them.

• The relevant data does not exist in the database. When the data does not exist, it’s impossible to retrieve it. Many retrieval techniques, however, give a relevance score to each result that they return, so filtering out low-relevance snippets may help mitigate the issue.

• The retrieval algorithm is too naive to match a question with its relevant context. This is a common issue for keyword-based retrieval methods such as TF-IDF or BM25 (Elasticsearch). These algorithms can’t deal with synonims or resolve acronyms, so if the question and the relevant context don’t share the exact same words, the retrieval won’t work.

• Embedding model (if used) is too small or unsuitable for the data. The data must be embedded before being searchable when doing a vector-based search. “Embedded” means that every snippet of context is associated with a list of numbers called an embedding. The quality of the embedding then determines the quality of the retrieval. If you embed your documents with a naive embedding model, or if you are dealing with a very specific domain such as narrow medical and legal niches, the embedding of your data won’t be able to represent their content precisely enough for the retrieval to be successful.

• The data is not chunked properly (too big or too small chunks). Retrievers thrive on data that is chunked properly. Huge blocks of text will be found relevant to almost any question and will drown the LLM with information. Too small sentences or sentence fragments won’t carry enough context for the LLM to benefit from the retriever’s output. Proper chunking can be a huge lever to improve the quality of your retrieval.

• The data and the question are in different languages. Keyword-based retrieval algorithms suffer from this issue the most because keywords in different languages rarely match. If you expect questions to come in a different language than the data you are retrieving from, consider adding a translation step or performing retrieval with a multilingual embedder instead.

One caveat with retrieval failures is that if you’re using a very powerful LLM such as GPT-4, sometimes your LLM is smart enough to understand that the retrieved context is incorrect and will discard it, hiding the failure. This means that it’s even more important to make sure retrieval is working well in isolation, something we will see in a moment.

Generation failure

Assuming that retrieval was successful, what would happen if the LLM still hallucinated?

This is clearly an issue with our LLM: even when given all the correct data, the LLM still generated a wrong answer. Maybe our LLM doesn’t know that Esperanto is even a language? Or perhaps we’re using an LLM that doesn’t understand English well?

Naturally, each LLM will have different weak points that can trigger issues like these. Here are some common reasons why you may be getting generation failures.

• The model is too small and can’t follow instructions well. When building in a resource-constrained environment (such as local smartphone apps or IoT), the choice of LLMs shrinks to just a few tiny models. However, the smaller the model, the less it will be able to understand natural language, and even when it does, it limits its ability to follow instructions. If you notice that your model consistently doesn’t pay enough attention to the question when answering it, consider switching to a larger or newer LLM.

• The model knows too little about the domain to even understand the question. This can happen if your domain is highly specific, uses specific terminology, or relies on uncommon acronyms. Models are trained on general-purpose text, so they might not understand some questions without finetuning, which helps specify the meaning of the most critical key terms and acronyms. When the answers given by your model somewhat address the question but miss the point entirely and stay generic or hand-wavy, this is likely the case.

• The model is not multilingual, but the questions and context may be. It’s essential that the model understands the question being asked in order to be able to answer it. The same is true for context: if the data found by the retriever is in a language that the LLM cannot understand, it won’t help it answer and might even confuse it further. Always make sure that your LLM understands the languages your users use.

• The RAG prompt is not built correctly. Some LLMs, especially older or smaller ones, may be very sensitive to how the prompt is built. If your model ignores part of the context or misses the question, the prompt might contain contradicting information, or it might be simply too large. LLMs are not always great at finding a needle in the haystack: if you are consistently building huge RAG prompts and you observe generation issues, consider cutting it back to help the LLM focus on the data that actually contains the answer.

Evaluation strategies

Once we put our RAG system in production, we should keep an eye on its performance at scale. This is where evaluation frameworks come into play.

To properly evaluate the performance of RAG, it’s best to perform two evaluation steps:

1. Isolated Evaluation. Being a two-step process, failures at one stage can hide or mask the other, so it’s hard to understand where the failures originate from. To address this issue, evaluate the retrieval and generation separately: both must work well in isolation.

2. End to end evaluation. To ensure the system works well from start to finish, it’s best to evaluate it as a whole. End-to-end evaluation brings its own set of challenges, but it correlates more directly to the quality of the overall app.

Evaluating Retrieval

Each retrieval method has its own state-of-the-art evaluation method and framework, so it’s usually best to refer to those.

For keyword-based retrieval algorithms such as TD-IDF, BM25, PageRank, and so on, evaluation is often done by checking the keywords match well. For this, you can use one of the many metrics used for this purpose: recall, precision, F1, MRR, MAP, …

For vector-based retrievers like vector DBs, the evaluation is more tricky because checking for matching keywords is not sufficient: the semantics of the question and the answer must evaluated for similarity. We are going to see some libraries that help with this when evaluating generation: in short, they use another LLM to judge the similarity or compute metrics like semantic similarity.

Evaluating Generation

Evaluating an LLM’s answers to a question is still a developing art, and several libraries can help with the task. One commonly used framework is UpTrain, which implements an “LLM-as-a-judge” approach. This means that the answers given by an LLM are then evaluated by another LLM, normally a larger and more powerful model.

This approach has the benefit that responses are not simply checked strictly for the presence or absence of keywords but can be evaluated according to much more sophisticated criteria like completeness, conciseness, relevance, factual accuracy, conversation quality, and more.

This approach leads to a far more detailed view of what the LLM is good at and what aspects of the generation could or should be improved. The criteria to select depend strongly on the application: for example, in medical or legal apps, factual accuracy should be the primary metric to optimize for, while in customer support, user satisfaction and conversation quality are also essential. For personal assistants, it’s usually best to focus on conciseness, and so on.

💡 UpTrain can also be used to evaluate RAG applications end-to-end. Check its documentation for details.

End-to-end evaluation

The evaluation of RAG systems end-to-end is also quite complex and can be implemented in many ways, depending on the aspect you wish to monitor. One of the simplest approaches is to focus on semantic similarity between the question and the final answer.

A popular framework that can be used for such high-level evaluation is RAGAS. In fact, RAGAS offers two interesting metrics:

• Answer semantic similarity. This is computed simply by taking the cosine similarity between the answer and the ground truth.

• Answer correctness. Answer correctness is defined as a weighted average of the semantic similarity and the F1 score between the generated answer and the ground truth. This metric is more oriented towards fact-based answers, where F1 can help ensure that relevant facts such as dates, names, and so on are explicitly stated.

On top of evaluation metrics, RAGAS also offers the capability to build synthetic evaluation datasets to evaluate your app against. Such datasets spare you the work-intensive process of building a real-world evaluation dataset with human-generated questions and answers but also trade high quality for volume and speed. If your domain is very specific or you need extreme quality, synthetic datasets might not be an option, but for most real-world apps, such datasets can save tons of labeling time and resources.

💡 RAGAS can also be used to evaluate each step of a RAG application in isolation. Check its documentation for details.

💡 I recently discovered an even more comprehensive framework for end-to-end evaluation called continuous-eval from Relari.ai, which focuses on modular evaluation of RAG pipelines. Check it out if you’re interested in this topic and RAGAS doesn’t offer enough flexibility for your use case.

Putting it all together

Once you know how you want to evaluate your app, it’s time to put it together. A convenient framework for this step is Haystack, a Python open-source LLM framework focused on building RAG applications. Haystack is an excellent choice because it can be used through all stages of the application lifecycle, from prototyping to production, including evaluation.

Haystack supports several evaluation libraries including UpTrain, RAGAS and DeepEval. To understand more about how to implement and evaluate a RAG application with it, check out their tutorial about model evaluation here.

Advanced flavors of RAG

Once our RAG app is ready and deployed in production, the natural next step is to look for ways to improve it even further. RAG is a very versatile technique, and many different flavors of “advanced RAG” have been experimented with, many more than I can list here. Depending on the situation, you may focus on different aspects, so let’s list some examples of tactics you can deploy to make your pipeline more powerful, context-aware, accurate, and so on.

Use multiple retrievers

Sometimes, a RAG app needs access to vastly different types of data simultaneously. For example, a personal assistant might need access to the Internet, your Slack, your emails, your personal notes, and maybe even your pictures. Designing a single retriever that can handle data of so many different kinds is possible. Still, it can be a real challenge and require, in many cases, an entire data ingestion pipeline.

Instead of going that way, you can instead use multiple retrievers, each specialized to a specific subset of your data: for example, one retriever that browses the web, one that searches on Slack and in your emails, one that checks for relevant pictures.

When using many retrievers, however, it’s often best to introduce another step called reranking. A reranker double-checks that all the results returned by each retriever are actually relevant and sorts them again before the RAG prompt is built. Rerankers are usually much more precise than retrievers in assessing the relative importance of various snippets of context, so they can dramatically improve the quality of the pipeline. In exceptional cases, they can be helpful even in RAG apps with a single retriever.

Here is an example of such a pipeline built with Haystack.

Self-correction

We mentioned that one of the most common evaluation strategies for RAG output is “LLM-as-a-judge”: the idea of using another LLM to evaluate the answer of the first. However, why use this technique only for evaluation?

Self-correcting RAG apps add one extra step at the end of the pipeline: they take the answer, pass it to a second LLM, and ask it to assess whether the answer is likely to be correct. If the check fails, the second LLM will provide some feedback on why it believes the answer is wrong, and this feedback will be given back to the first LLM to try answering another time until an agreement is reached.

Self-correcting LLMs can help improve the accuracy of the answers at the expense of more LLM calls per user question.

Agentic RAG

In the LLMs field, the term “agent” or “agentic” is often used to identify systems that use LLMs to make decisions. In the case of a RAG application, this term refers to a system that does not always perform retrieval but decides whether to perform it by reading the question first.

For example, imagine we’re building a RAG app to help primary school children with their homework. When the question refers to topics like history or geography, RAG is very helpful to avoid hallucinations. However, if the question regards math, the retrieval step is entirely unnecessary, and it might even confuse the LLM by retrieving similar math problems with different answers.

Making your RAG app agentic is as simple as giving the question to an LLM before retrieval in a prompt such as:

Reply YES if the answer to this question should include facts and 
figures, NO otherwise.

Question: What's the capital of France?

Then, retrieval is run or skipped depending on whether the answer is YES or NO.

This is the most basic version of agentic RAG. Some advanced LLMs can do better: they support so-called “function calling,” which means that they can tell you exactly how to invoke the retriever and even provide specific parameters instead of simply answering YES or NO.

For more information about function calling with LLMs, check out OpenAI’s documentation on the topic or the equivalent documentation of your LLM provider.

Multihop RAG

Multihop RAG is an even more complex version of agentic RAG. Multihop pipelines often use chain-of-thought prompts, a type of prompt that looks like this:

You are a helpful and knowledgeable agent.

To answer questions, you'll need to go through multiple steps involving step-by-step 
thinking and using a search engine to do web searches. The browser will respond with 
snippets of text from web pages. When you are ready for a final answer, respond with 
`Final Answer:`.

Use the following format:

- Question: the question to be answered
- Thought: Reason if you have the final answer. If yes, answer the question. If not, 
    find out the missing information needed to answer it.
- Search Query: the query for the search engine
- Observation: the search engine will respond with the results
- Final Answer: the final answer to the question, make it short (1-5 words)

Thought, Search Query, and Observation steps can be repeated multiple times, but 
sometimes, we can find an answer in the first pass.

---

- Question: "Was the capital of France founded earlier than the discovery of America?"
- Thought: 

This prompt is very complex, so let’s break it down:

1. The LLM reads the question and decides which information to retrieve.

2. The LLM returns a query for the search engine (or a retriever of our choice).

3. Retrieval is run with the query the LLM provided, and the resulting context is appended to the original prompt.

4. The entire prompt is returned to the LLM, which reads it, follows all the reasoning it did in the previous steps, and decides whether to do another search or reply to the user.

Multihop RAG is used for autonomous exploration of a topic, but it can be very expensive because many LLM calls are performed, and the prompts tend to become really long really quickly. The process can also take quite some time, so it’s not suitable for low-latency applications. However, the idea is quite powerful, and it can be adapted into other forms.

A word on finetuning

It’s important to remember that finetuning is not an alternative to RAG. Finetuning can and should be used together with RAG on very complex domains, such as medical or legal.

When people think about finetuning, they usually focus on finetuning the LLM. In RAG, though, it is not only the LLM that needs to understand the question: it’s crucial that the retriever understands it well, too! This means the embedding model needs finetuning as much as the LLM. Finetuning your embedding models, and in some cases also your reranker, can improve the effectiveness of your RAG by orders of magnitude. Such a finetune often requires only a fraction of the training data, so it’s well worth the investment.

Finetuning the LLM is also necessary if you need to alter its behavior in production, such as making it more colloquial, more concise, or stick to a specific voice. Prompt engineering can also achieve these effects, but it’s often more brittle and can be more easily worked around. Finetuning the LLM has a much more powerful and lasting effect.

Conclusion

RAG is a vast topic that could fill books: this was only an overview of some of the most important concepts to remember when working on a RAG application. For more on this topic, check out my other blog posts and stay tuned for future talks!