Deploy Verify

Your build passed. That doesn’t mean it worksphoto. Deploy Verify uses AI-powered browser agents to test real user flows after every deploy—so you catch failures before users do.

Comment

Inspiration

Deploy Verify came out of a recurring gap I kept hitting in my own deployment workflows.

Modern CI pipelines are very good at answering the question: “Did the build succeed?”

They are much worse at answering: “Does the site actually work for a user right now?”

Unit tests, integration tests, and even traditional browser tests often verify implementation details rather than outcomes. They rely on fragile selectors, tightly coupled assumptions, and test code that ages poorly as UIs evolve. As a result, the final check—opening the production site and clicking around manually—still happens far too often.

I wanted a tool that runs after deployment, from the outside, and validates behavior the same way a human would. If a user can’t log in, submit a form, or load the dashboard, the deployment should be considered suspect—regardless of how many tests passed earlier.

Discovering Nova Act made me realize I could finally build the type of solution that would solve this long-standing sore spot in my workflow. Post-deployment verification using the power of Nova Act looks less like writing code and more like describing what a user does.

As AI coding increasingly becomes the way we create apps, using human language—not machine language—to verify a build becomes more important. That’s exactly what Deploy Verify brings to anyone building modern web applications.

What it does

Deploy Verify allows developers to create one or more post-deployment tests for a live site using natural language.

Tests can be written manually in structured plain English or generated with the built-in AI Writing Assistant powered by Nova 2 Lite, which takes a free-form description and rewrites it into clear, step-by-step instructions and assertions.

Each site receives a unique webhook that can be added to a deployment pipeline. When a deployment completes and calls the webhook, Deploy Verify uses Nova Act to launch a real browser session and execute the tests against the live site.

After execution, the developer receives a complete verification report that includes:

  • A natural-language summary of results (generated by Nova 2 Lite)
  • Screenshots captured during the test
  • The full HTML source of relevant pages
  • JavaScript console logs
  • Detailed step-by-step pass/fail results

Deploy Verify is designed to be most valuable when tests fail.

When a failure occurs, the system provides Nova 2 Lite with:

  • The failed test instructions and assertions
  • The structured test results
  • Artifacts from the browser like HTML source code and Javascript error logs
  • Plus, every Git commit message since the last successful deployment

Using this context, the AI analyzes the failure, reasons about likely causes, and suggests possible solutions—turning a failed deployment into actionable insight rather than a generic error state.

How I built it

I turned to one of my favorite frameworks, Laravel, to build the project. I started with a basic proof-of-concept app to prove to myself that the promise of Deploy Verify actually delivered. I had a simple version running a Nova Act test triggered by a webhook with hard-coded instructions. After those tests worked, I dove in and added additional features to make the user experience better and more useful in real-world scenarios.

Beyond Nova Act for browser automation, the app uses Amazon Bedrock (via the Converse API) to power three AI-driven features: an instruction helper that converts plain-English test descriptions into optimized Nova Act instructions and assertions, a failure diagnosis service that analyzes failed runs — including HTML source, console logs, and recent commit history — to pinpoint the likely cause and suggest a fix, and a run summary generator that produces concise overviews of verification results. All three services call Amazon Nova 2 Lite through Bedrock, keeping the AI layer fully within the AWS ecosystem alongside Nova Act.

Two of those features are worth a closer look.

AI-Assisted Test Creation

Early testers showed me that even structured natural language can feel intimidating to first-time users. To reduce friction, I added an AI Helper powered by Nova 2 Lite.

Users can describe their test in loose, conversational language, and the AI generates an initial set of instructions and assertions. The result is a test that feels approachable while remaining deterministic and editable.

Making failed tests more useful

An early failed deployment test inspired me to focus on making failed tests the reason someone would come to rely on Deploy Verify. If I provided enough context, Nova 2 Lite could help diagnose and fix failed tests.

Tests already generate screenshots, HTML snapshots, and console logs. I also include the commit messages from every build since the last successful run, along with the data I already collect, to give Nova 2 Lite more context and generate more actionable suggestions when deployment tests fail.

Challenges I ran into

An unexpected challenge was constraining natural language without making it feel restrictive. Instructions had to be flexible enough to feel human, yet structured enough to run deterministically.

Multiple iterations on phrasing, validation, and best practices led directly to the AI writing assistant mentioned above. That piece makes Deploy Verify much easier to set up for each site you want to test.

The other major challenge was securing the browser process itself. Deploy Verify runs a real browser on behalf of its users — which means a malicious user could craft instructions that turn the browser against the server. Without safeguards, the browser could reach internal services, hit the AWS metadata endpoint, or probe the local network.

I tackled this with defense in depth: URL validation blocks internal IP ranges and dangerous schemes before a test even starts, iptables rules restrict the browser's network access at the kernel level, and the browser runs as a dedicated low-privilege OS user (dv-browser) that has no access to application code, secrets, or SSH keys. I also inject JavaScript hardening into every page to stub out window.open() and restrict navigation to safe schemes. Each layer catches what the others might miss.

Accomplishments that I'm proud of

I’m proud of how tightly the AI tools are integrated into the app. Before discovering Nova Act, I could have built tests using regular expressions and other hard-coded processes, but that kind of brittle testing is exactly what caused problems for me in the past. Deploy Verify is an AI-first solution, not an app with AI tacked on.

I’m also proud of the overall finish of the app. I’ve spent significant time refining the UX to make using it feel natural and intuitive. If you visit the Deploy Verify homepage, I hope you find it to be as polished as any SaaS app you've tried.

What I learned

AI is dramatically accelerating how fast code is written and deployed, but verification hasn’t kept pace. As AI-generated code becomes more common, developers are shipping changes they didn’t handcraft line by line.

In that world, verifying correctness through selectors, DOM structure, or internal assumptions breaks down quickly. What still matters is behavior: can a user log in, complete a flow, and reach the right outcome?

As software creation velocity increases, tools that verify outcomes instead of implementation details become essential rather than optional.

What’s next for Deploy Verify

The next step is moving Deploy Verify from a hackathon project into a production-ready service.

I’ve already built an invite system so I can onboard early users gradually. The immediate goal is to get the tool into real deployment pipelines and let real-world usage reveal where it needs to evolve. Production sites introduce complexity—authentication flows, timing issues, and unpredictable UI changes—that will help refine how Deploy Verify interprets and executes tests with Nova Act.

Another major focus is expanding failure intelligence.

Deploy Verify already analyzes failed tests using browser artifacts and recent Git commit messages. In early experiments, commit messages have proven to be one of the most useful signals for understanding why a deployment failed.

Because Deploy Verify is designed for the era of AI-assisted coding, there’s an opportunity to push this even further. Future versions could provide developers with a small “skill” or instruction set for their AI coding agents that encourages writing commit messages optimized for debugging and deployment analysis. By shaping the commit history intentionally, Deploy Verify gains better context when failures occur—making its diagnosis significantly more useful.

Longer term, the goal is to make post-deployment verification a standard part of modern CI pipelines.

CI systems answer “Did the build succeed?” Deploy Verify answers the question that actually matters: “Does the deployed application work for a user right now?”

If successful, Deploy Verify becomes the final safety layer in AI-accelerated software development—an automated user checking the site after every deployment.

Share this project:

Updates