We saw this during a pilot where an LLM-based assistant was used to generate regression tests for an e-commerce checkout service.

The workflow looks like this:

1. Developer submits PR with changes to discount calculation logic.
2. CI triggers test generation using an LLM.
3. Generated tests are added to a review queue.
4. High-confidence tests are merged automatically.

At first, it worked well. Coverage improved. Edge cases increased. Regression escapes decreased.

Six weeks later, a production defect appeared: orders combining a gift card with a percentage-off coupon were calculating the wrong final charge. The defect should have been caught.

Using the layered taxonomy, the diagnosis becomes clearer.

Layer 1 – Model Behaviour: The prompt asked for “discount logic edge cases.” The model generated structurally valid tests but consistently used assertions that checked status codes rather than calculated values for combined discount scenarios. The gift card plus coupon path was tested. Whether the amount was correct was never asserted. Not incorrect. Just systematically shallow on what mattered.

Failure pattern: Output inconsistency + assertion gap on compound state logic.

Layer 2 – Retrieval and context: Historical bug reports were included as retrieval context. However, retrieval ranking prioritised recent high-frequency issues. A year-old bug about combined discount miscalculations was ranked low and truncated before reaching the context window. The model never saw the relevant historical signal.

Failure pattern: Recency bias + context truncation of low-frequency high-severity bugs.

Layer 3 – Orchestration: The CI workflow auto-approved tests when overall coverage crossed 80%. The new tests pushed coverage from 78% to 83%, but all the new coverage landed on already well-tested single-discount paths. The threshold was satisfied. Tests were merged. Nothing checked which paths the additional coverage actually represented.

Failure pattern: Threshold passed without coverage quality check.

Layer 4 – Human and trust: Over time, the QA engineer stopped reading individual test cases and checked only the coverage percentage on the dashboard. “Numbers have looked fine for weeks.” Manual validation depth declined.

Failure pattern: Metric substitution + confidence drift at the human layer.

The variability above focuses on single-model behaviour. Modern AI architecture extends this complexity further.

Retrieval-Augmented Generation (RAG) systems combine probabilistic model outputs with deterministic retrieval logic. In practice, these outcomes are influenced by very ordinary engineering decisions: chunk size selection, retrieval recall versus precision trade-offs, embedding consistency challenges as the corpus evolves, and hybrid search strategies that rebalance semantic and keyword ranking.

Output quality now depends on:

• Model interpretation of the query
• Retrieval ranking and document selection
• Data freshness and indexing
• Context window constraints
• Response synthesis

Each layer introduces its own failure modes.

A model may generate a coherent response based on incomplete retrieval results. The Retrieval quality can degrade quietly as documents are added, re-indexed, or re-embedded. Outdated documents may remain indexed. Every component can behave “correctly” in isolation while the system-level outcome is unacceptable.

Failures now emerge from orchestration, not just individual defects.

As AI systems evolve toward agents that orchestrate tools and multi-step workflows, the interaction surface expands further.

This is where deterministic testing assumptions fail structurally. Verification-only approaches cannot detect failures that emerge from component interaction rather than component defects.

If we continue to apply deterministic testing models to probabilistic systems, we are systematically under-testing AI in production.