TestFlows.com Open-Source Testing Framework

🎯 Every Test Has a Purpose — Are You Tracking It? The only true purpose of writing any test is to verify one or more requirements. Yet most teams skip them entirely. TestFlows has supported structured requirements in Markdown linked directly to tests long before AI became mainstream. Now, with AI in the picture, that decision looks better than ever. AI can help you write requirements, and once they exist as structured Markdown files versioned in Git, AI can use them to generate, validate, and maintain your tests. No more excuses! https://lnkd.in/gbVd9Kbi

🍄 Autonomous testing is great, and when you combine it with a behavior model, it gets even better (https://lnkd.in/gDjXkyUg). In Part 2 of our Super Mario Bros. autonomous testing series , we take autonomous state-space exploration and connect it to our behavior model that validates Mario’s movements frame by frame during gameplay. Curious how the behavior model looks? Take a peek: https://lnkd.in/gQYV2vWY. Building on the work from Part 1—where we focused on developing an algorithm to autonomously conquer levels—Part 2 asks a deeper question: is Mario’s behavior actually valid? To answer it, we plug in a behavior model that continuously checks every frame against safety, causal, and liveness properties describing how Mario’s movements should behave. The result: we can detect impossible movements, invalid physics, unexpected teleports, and other engine anomalies — with deterministic replay to reproduce every issue exactly. Key takeaways: 🛡 Safety checks — detect overlapping tiles, illegal positions, and broken collision responses. 🔗 Causal validation — movements must follow from real inputs or engine-driven effects. ⚖️ Bidirectional refinement — violations reveal both game-engine bugs and gaps in the model. ⏱ Deterministic replay — every incorrect frame is fully reproducible for debugging. This turns autonomous exploration into autonomous behavior verification — checking not only that Mario reaches the goal, but how he does it. If you missed Part 1: https://lnkd.in/gNsmiiSm 🧑💻 All code is open-source! Clone: https://lnkd.in/giFy2zRB Run: python3 tests/run.py --autonomous --play-seconds 300 👽 Ready to go beyond tests and start writing test programs? Try the open-source TestFlows.com Open-Source Testing Framework: https://testflows.com #AutonomousTesting #BehaviorModels #SoftwareTesting #GameTesting #TestFlowsOSTF

🛸 There are only two hard problems in testing: 🔹The size of the state space  🔹Knowing whether the state is valid Everything else you can think of flows from these two. State space → too many possible situations to cover exhaustively (combinatorial explosion).  State validity → even when you reach a situation, you need a clear way to decide correctness (the test oracle problem). If you’ve ever wondered when to stop testing and felt you could never do enough — that’s the state-space problem. If you’ve ever struggled to tell whether the result is correct after thousands of random actions — that’s the oracle problem. There’s no free lunch: in the general case — and in most non-trivial projects — both defy complete solutions. Still, autonomous testing and behavior models are the best tools for doing our best in the face of them. Deep dive into autonomous testing and behavior models in our series:  👉 https://lnkd.in/gkRHH-xM  👉 https://lnkd.in/gNsmiiSm Or just play with the code:  Clone: https://lnkd.in/gRweDjUs  Run it: `python3 tests/run.py --autonomous --play-seconds 300` 👽 Ready to go beyond tests and start writing test programs?  Try the open-source TestFlows.com Open-Source Testing Framework: https://testflows.com #AutonomousTesting #BehaviorModelTesting #TestAutomation #SoftwareTesting #TestFlowsOSTF

🍄 Autonomous testing is trending. Learn how it works (https://lnkd.in/gNsmiiSm) with an open-source Super Mario Bros. example in our Testing Super Mario Using a Behavior Model Autonomously (Part 1) blog post. Inspired by the Antithesis post “How Antithesis finds bugs (with help from the Super Mario Bros.)”(https://lnkd.in/g8jmqR2j), we share a complete and practical implementation you can run yourself. We implement Super Mario’s autonomous state-space exploration through a continuous select → replay → mutate → evaluate loop — using weighted mutation-based input generation that blends random fuzzy mutations with structured move patterns, a probabilistic path-selection strategy, and deterministic replay for reproducibility. The result: Mario learns to conquer levels entirely on his own, no human guidance required — scoring progress by level, distance, and time while uncovering interesting behaviors along the way. Key takeaways: 🧬 Autonomous exploration — weighted mutation and probabilistic selection drive Mario’s progress through the game world. ⚙️ Genetic Algorithms in action — population, selection, and mutation principles applied to real gameplay testing. ⏱ Determinism + instrumentation — accelerate exploration with reproducible runs and high-speed replay. 🎮 Levels conquered — watch videos of our autonomous agent completing all four levels of Super Mario Bros. Stay tuned for Part 2, where we’ll integrate autonomous testing with our previously developed behavior model to validate Mario’s actions frame by frame, verifying not only that he can win but also that he wins correctly. 🧑💻 All the code is open-source! Clone: https://lnkd.in/gRweDjUs Run it: python3 tests/run.py --autonomous --play-seconds 300 👽 Ready to go beyond tests and start writing test programs? Try the open-source TestFlows framework: https://testflows.com #AutonomousTesting #GeneticAlgorithms #SoftwareTesting #GameTesting #OpenSource #PythonTesting #AIinTesting #OpenSourceTesting #TestFlowsOSTF

🍄 Behavior models take property-based testing to the next level — and in Part 2 (https://lnkd.in/gzBkYhkC) we implement a simple executable model that verifies Super Mario’s behavior using different forms of safety properties — causal, classical, and bounded liveness. In Part 1, we built the test plumbing and explored the limits of end-to-end tests. In Part 2, we now show how to express correctness using properties that check every frame of the game: 🧩 Causal properties — every movement must have a valid cause 🛡️ Classical safety properties — nothing impossible ever happens ⚡ Bounded liveness properties — expected actions eventually occur within a defined time window Together, they form a composable, incremental model that validates behavior continuously — frame by frame — allowing Mario’s movement to be verified continuously throughout gameplay. 📘 Part 2: https://lnkd.in/gzBkYhkC 👀 Part 1 (recap): https://lnkd.in/gkRHH-xM 🧑💻 Find out for yourself — all the code is open-source: Source: https://lnkd.in/gCTbdX-5 Add your properties: https://lnkd.in/gZDUs6Fz Play & verify — run this for 5 minutes to see if your properties hold! ./tests/run.py --only "/super mario/manual/play/*" --manual-play-seconds=300 👽 Ready to start writing test programs instead of just tests? Use the TestFlows.com Open-Source Testing Framework: https://testflows.com #BehaviorModels #PropertyBasedTesting #ModelBasedTesting #SoftwareTesting #GameTesting #TestFlowsOSTF

🍄 Behavior models take property-based testing to the next level. In our new series (https://lnkd.in/gkRHH-xM), "Testing Super Mario Using a Behavior Model," we take a deep dive into the why and how. Inspired by the Antithesis post "How Antithesis finds bugs (with help from the Super Mario Bros.)" (https://lnkd.in/g8jmqR2j) — we ask a fundamental question: How do you verify the game behaves correctly in all explored states beyond a few basic properties? To answer this, in Part 1, we build the test plumbing from the ground up, implement classic end-to-end tests, and hit the fundamental challenges of testing complex stateful systems. In Part 2, we introduce the theory and implement an executable model to verify Mario's movements that can be plugged into any test driver — manual, automated, autonomous, or AI-driven. Throughout this journey, you'll learn that testing a game like Super Mario is way harder than it seems! 📘 Part 1: https://lnkd.in/gkRHH-xM 👀 Part 2 (sneak peek): https://lnkd.in/gzBkYhkC 🧑💻 Get your hands dirty — all the code is open-source: Source: https://lnkd.in/ge7PWerc Add your properties: https://lnkd.in/gD5sqAHH Play & verify: Run this for 5 minutes to see if your properties hold! ./tests/run.py --only "/super mario/manual/play/*" --manual-play-seconds=300 👽 Ready to start writing test programs instead of just tests? Use the TestFlows.com Open-Source Testing Framework: https://testflows.com #PropertyBasedTesting #BehaviorModels #ModelBasedTesting #SoftwareTesting #GameTesting #TestFlowsOSTF

Vibe coding isn’t just about turning plain English into code. Can AI understand math even better? For fun, we ran a simple experiment. We started with the simple formal transition relation R for a Ceiling Speed Monitoring Controller from the 2013 paper Exhaustive Model-Based Equivalence Class Testing by Huang & Peleska (https://lnkd.in/gS9nAFds). Then we fed this small mathematical spec directly to ChatGPT 5 Thinking and Gemini 2.5 Pro to see if they could generate a correct-by-construction CLI—without the ambiguity of a prose description. The result: a tiny, functional CLI that matched the spec, one-shot. This suggests a workflow even more exciting than typical vibe coding. It points to a future where using AI we can: ‣ Generate code directly from formal specifications, leveraging the precision of math for higher reliability. ‣ Help develop the formal spec itself (identify states, outputs, guards, atomic propositions). ‣ Verify implementations using formal test programs built from those specs. It shows AI isn’t limited to plain-language specs; it can consume and produce from formal ones just as well—and perhaps with more predictable results. How was it verified? Not with head-scratching, but with a straightforward, model-based workflow derived directly from the spec: Formal Math → Atomic Propositions (AP) → Input Equivalence Class Partitions (IECP) → Test Program with Behavior Model → ✅ Curious to see the details? ‣ Code + Notes (Gist): https://lnkd.in/g2gEKcYv ‣ Spec walkthrough (transition relation R): https://lnkd.in/gzDjYKy4 ‣ Test program development: https://lnkd.in/gK5Zm56k 👽 Ready to start writing test programs instead of just tests? Use TestFlows.com Open-Source Testing Framework: https://testflows.com #VibeCoding #TestFlowsOSTF #ChatGPT #Gemini #FormalMethods #ModelBasedTesting #AI #SoftwareTesting #SoftwareQualityAssurance

🚀 Stop writing just tests. Start writing test programs. Most frameworks are runner-centric → you write tests, the runner controls everything. But what if you need more? Conditional execution, loops with dynamic parameters, parallel flows, combinatorics, behavior models… That’s where test programs change the game. 📌 You write tests and control flow in code. 📌 You scale from simple checks to advanced models. 📌 You unify your approach instead of juggling multiple tools. This isn’t theory. It’s how you can test with TestFlows.com Open-Source Testing Framework. 👉 Read the full article here: https://lnkd.in/giBvuzAN. #SoftwareTesting #TestAutomation #Python #OpenSource #TestFlowsOSTF #SoftwareQualityAssurance #TestPrograms

We cached 187 GB of Docker images in GitHub Actions using self-hosted runners — delivering 3× speedup. Just tested our demo workflow (https://lnkd.in/gd7-nwQt) that pulled 93 popular Docker images with our TestFlows Hetzner Runners (https://lnkd.in/g37w8R5s). 📊 Results: ▸ Fresh run (no cache): 19m 07s → Run (https://lnkd.in/gu6emJFN) ▸ Cached run: 4m 02s → Run (https://lnkd.in/gBK83xGa) On the first run, Docker synced 187 GB to the caching volume in 6m 18s — an effective throughput of ~495 MB/s (~4 Gbps) on Hetzner storage. 💡 This means: ▸ Cache 187 GB once ▸ Never pull the exact same image again ▸ Enjoy 3× faster runs 👉 Try it yourself: Using Caching Volumes (https://lnkd.in/gd3cGETH) Ready to start writing test programs instead of just tests? Explore the TestFlows.com Open-Source Testing Framework. #GitHubActions #CI #CICD #DevOps #Hetzner #Docker #SelfHostedRunners #TestFlows #TestFlowsOSTF

🔥 Persistent caching volumes are now available in TestFlows GitHub Hetzner Runners! Attach up to 16 volumes per runner, each from 10 GB to 10 TB — that’s almost unlimited caching for your CI pipelines. Speed up and add stability by reusing data across jobs and workflows — no hacks, no limitations. Flexible and easy to use — just add a label to your jobs. Want faster Docker pulls and builds? Start here 👉 https://lnkd.in/gab4QCEP ✨ Highlights:    🔄 Share data across jobs and workflows with persistent volumes    🐧 Full root access on real VMs    🐳 Native Docker & docker-compose — no tricks, just real Docker    🚫 No AWS spot instance surprises    💸 Up to 80× cheaper than GitHub-hosted runners    ⚙️ Ideal for caching Docker layers, wheels, APT packages & more Learn how to enable persistent caching: https://lnkd.in/gd3cGETH 👽 Ready to start writing test programs instead of just tests? Explore TestFlows.com Open-Source Testing Framework for all your testing needs: https://testflows.com #GitHubActions #CI #CICD #DevOps #Hetzner #Docker #SelfHostedRunners #TestFlows #TestFlowsOSTF