1. You have way more AI than you think

It’s not just what engineering deployed.

• Marketing tools

• Sales integrations

• Hidden vendor features

Your AI attack surface is already bigger than your inventory.

If you don’t map it first, everything else breaks.

2. “High / Medium / Low” risk is useless

AI (or any) risk doesn’t fit clean labels.

Everything becomes “medium.”

The shift that works?

Quantify risk in dollars

When you do that:

• You can justify controls

• You can prioritize correctly

• Leadership actually engages

3. Overly strict policies create Shadow AI

“Don’t create IP risk with AI” sounds good.

But what does it actually mean?

The companies that win:

• Involve the business

• Create usable policies

• Enable adoption within risk appetite

4. The standard tells you what, not how

ISO 42001 says:

“Do impact assessments.”

It doesn’t tell you:

• What questions to ask

• How to scope them

• What auditors expect

This is where most DIY efforts die.

5. Security shouldn’t own risk decisions

Security advising ≠ security deciding.

If security owns risk everything gets blocked (and logically, should)

The model that works:

• Security advises

• Business decides

• And owns accountability for outcomes

6. Certification is not the finish line

Most programs drift within 6 months.

Why?

• Models update

• Vendors change

• Regulations shift

If your system isn’t continuously updated:

Your certification becomes fiction.

7. The ROI is there

Yes, it reduces regulatory risk.

But the real upside:

• Faster enterprise sales

• Shorter security reviews

• Clear answers during procurement

Bottom line

Many approach ISO 42001 like a compliance project.

The ones that succeed treat it like a business system for scaling AI securely, safely, and responsibly.

Need help getting ISO 42001 ready?

Book a call

Slack just overrode your settings, activating AI features.

Previously they gave admins a heads up, but looks like this time they decided to ask for forgiveness rather than permission.

Here’s the full timeline and a comparison with how they previously did AI feature enablement:

The July 2025 update

On July 8, 2025, I got a “July Admin Update” from Slack with the below:

This linked to this page (now expired, but captured with Archive.org and with a screenshot below):

Based on this warning, I disabled the AI settings on this menu (and screenshotted it):

I wasn’t ready to roll out the AI features and didn’t need them, so turned them off.

March 2026 surprise

On March 12, I opened slack and saw this notification:

And when you navigate to the Admin menu, saw a new set of toggles. Originally “AI Search” and “AI filters (Beta)” were defaulted to “Everyone can use.”

Because my position hadn’t changed about Slack AI, I turned them off. Even later on the 12th when trying to recreate this screen, it had already changed.

Concerned about the lack of heads up, I checked the “February Admin update” (sent February 11, 2026) and “March Admin update” (sent March 9, 2026) there was no mention of the coming AI feature activation.

AI is moving fast

I’m not an AI security alarmist, and this isn’t likely to change your data security posture.

But this likely violates ISO 42001 change control processes. The key requirements for here are from paragraph 8.1, requiring the organization to:

• “control planned changes and review the consequences of unintended changes, taking action to mitigate any adverse effects, as necessary.

• “ensure that externally provided processes, products or services that are relevant to the AI management system are controlled.”

And I’m not happy about the auto-enabling of “Preview” AI search and “Beta” AI filters.

I can also definitely tell you Admins are NOT in control (unless they go in and disable after the fact).

So going forward, I’m:

• Tracking this event as a non-conformity.

• Documenting “Auto-enabling of AI features” as a risk in my register.

• Exploring ways to automatically monitor for the activation of AI features.

An ISO 42001 audit can build customer trust and reduce regulatory risk, if you are prepared...

...which is why StackAware is partnering with Prescient Security to help AI-powered companies through the entire governance lifecycle.

How it works:

• StackAware builds defensible AI governance

• Controls exist in systems, not slide decks

• Evidence is continuous, not point-in-time

• Risk criteria is documented & defensible

• Prescient Security evaluates conformance

This matters because AI audits fail for the same reasons:

• Undefined scope

• Unclear definitions

• Inconsistent risk ownership

• Controls that exist only on paper

• Evidence scattered across teams

By the time an audit starts, fixing this costs time, money, and credibility.

Prescient Security sees these failures all the time.

That’s why this partnership exists.

“The biggest audit issues we see aren’t technical failures. They’re governance gaps. Teams know AI matters, but they haven’t operationalized decisions in a way that stands up to scrutiny. StackAware will help our customers close gaps before an audit begins.”

- Sammy Chowdhury, Co-Founder & Chief Compliance Officer , Prescient Security

So if you’re preparing for ISO 42001, StackAware gets you ready so audits don’t create surprises.

And to support Prescient Security clients, we’re offering:

One free month of StackAware’s AI Governance Advisor service ($499 value).

Claim it here

Risk transfer is key to comprehensive AI governance.

So I’m excited to partner with Armilla AI on AI insurance.

Together, we’re helping firms:

• Avoid unacceptable AI risk

• Mitigate what’s left with controls

• Accept risk when it makes business sense

• Transfer residual AI risk with purpose-built insurance

How the partnership works:

1. StackAware helps companies implement ISO/IEC 42001-aligned AI governance programs.

2. Armilla AI evaluates real-world system performance and failure modes to support underwriting.

3. Organizations demonstrating maturity unlock AI liability insurance on favorable terms.

This closes a major gap in the market.

Most companies today can talk about responsible AI.

Few can prove it in a way insurers, regulators, and enterprise customers accept.

By connecting ISO 42001 readiness with Armilla’s AI liability coverage, governance stops being a cost center and starts becoming a risk-financing lever.

Philip Dawson, Head of Partnerships at Armilla AI, said:

“Underwriting only works when you understand how systems behave in the real world. Partnering with StackAware allows us to translate strong AI governance into measurable risk signals—helping organizations both reduce loss exposure and transfer what remains.”

StackAware clients now have a clearer path to:

• Govern AI responsibly

• Validate it technically

• Insure it intelligently

If you’re deploying AI agents, models, or decision-support systems—and wondering how governance, testing, and insurance fit together—this is the link.

Want to learn more about how Armilla and StackAware are revolutionizing AI risk management?

Book a call

The explosion in use of AI-powered coding assistants represents a fundamental shift in software development. Leveraging them necessarily requires granting direct, high-privilege access to proprietary codebases. Some of our customers rely heavily on Cursor for both productivity and code intelligence - deep, context-aware assistance across the entire codebase. Given its deep integration with key workflows, we launched an ethical hacking campaign to validate its security.

This post, the first in our disSECt (dissect + security) series, details the methodology, technical struggles (including a time-intensive deep dive), and the actionable findings from that review.

Our aim is to provide a re-usable framework other engineering teams can leverage when auditing third-party tools with codebase access.

The target

Cursor is an AI-powered code editor/assistant. Our security review focused on its role as a local agent, its interaction with remote AI services, and, critically, the scope of its access to codebases.

We chose this tool because it has seen one of the highest adoption rates by our customers.

The process

Proxy

Due to Cursor’s foundation (specifically its integrated development environment [IDE] capabilities) on Visual Studio Code (VS Code), we used the native VS Code proxy settings for traffic interception and analysis.

To replicate this setup: Configure your Cursor instance by adjusting the settings.json (example location: C:\Users\user\AppData\Roaming\Cursor\User\settings.json) to the following values

{

    “window.commandCenter”: true,

    “http.proxyStrictSSL”: false,

    “http.experimental.systemCertificatesV2”: true,

    “cursor.general.disableHttp2”: true,

    “cursor.composer.shouldChimeAfterChatFinishes”: true,

    “cursor.diffs.useCharacterLevelDiffs”: true,

    “cursor.cpp.enablePartialAccepts”: true,

    “http.proxy”: “http://192.168.220.5:8080”,

    “http.electronFetch”: true,

    “http.proxyAuthorization”: null

}

After configuring the http.proxy variable to direct traffic to our inspection tool, we established visibility into the network layer.

Our initial findings showed that for request transport, Cursor relies on Hypertext Transfer Protocol Secure (HTTPS) carrying serialized data via Protocol Buffers (protobuf).

This architecture immediately presented a challenge for deep-traffic inspection and manual auditing. This is a security plus for Cursor because it makes it harder for an attacker to successfully analyze the system.

After evaluating several Burp Suite extensions for Protocol Buffer handling, we selected protobuf extensions from Google. Its human-readable output made the inspection process manageable.

See the example below:

While reviewing several web-based features, we discovered that the service would accept and process requests serialized in JSON instead of the native protobuf. This simple finding allowed us to bypass the complexity of protobuf decoding.

We could then conduct a detailed inspection of every user interface (UI) element and the entire underlying API surface.

Findings

Issue #1: Unintended cross-user access to custom documentation definitions

Imagine you are developing a new project and want Cursor’s agent to be aware of your project-specific documentation. Cursor addresses this business need with a predefined list of 3rd party project documentation available to the user.

This doesn’t solve another problem, however: this 3rd party documentation being outdated. Many open-source projects have documentation that is not up to date enough or needs internal project-specific adjustments. This is where the ability to add self-hosted documentation or custom Uniform Resource Locator (URLs) shine.

Cursors’ Docs library feature allows users to input arbitrary Uniform Resource Identifiers (URIs), including those leveraging the deprecated (RFC 3986) user information component (e.g., http://user:password@resource.com/path). This is a syntactically valid, though discouraged, URI format. After successful submission of the URL pointing to documentation you would like to add to the Cursor’s Doc’s library, the backend creates a unique ID unpredictable identification object representing the desired documentation.

This unique ID can have 3 forms. We assume the difference in format results from different sources, times at which the documentation was processed, and API versions were used to process the submission.

• First form: seems to be a sha256 hash

• Second form: a prefix+UUIDV4

• Third form: UUIDV4. UUIDv4 (universally unique identifier version 4) which looks like 20354d7a-e4fe-47af-8ff6-187bca92f3f9

Below you can find the example output server

In the graphic above you can observe the username a with password b that can be used to access the f00b4r.llamasbytes.com/wow resource. This documentation definition is represented by the ID that starts with f0d97ed characters. The ID itself is long and unpredictable.

Although the UI allows team members to explicitly select “Share with the team,” implying the documentation definition is not shared by default, we found out that the document’s definition remains accessible via its ID to other team members and users that are not part of the team.

After our inquiry, the vendor nonetheless described the feature as “working as intended.” That is because the supporting documentation of 3rd party projects is public anyway, even though it doesn’t always have to be, as we showed with the previous example). Hard to guess and random UUIDs are sufficient to mitigate Insecure Direct Object Reference (IDOR) risk, because if implemented correctly, an attacker can’t guess the values of IDs. Brute force attacks are difficult against the UUID due to the fact that the random part of a UUIDv4 has 122 random bits. Even if you could generate and check one trillion UUIDs per second, it would still take you billions of years to check a significant portion of the space.

Recommendation

Do not integrate documentation via URLs that contain embedded credentials in the URI. This increases the likelihood of sensitive data exposure, because the username and password part of URI might get exposed to 3rd party - and this will allow the 3rd party to access protected resources.

Issue #2: Undocumented default sharing of cloud agents granting unintended read access via GitHub repository permissions

In the age of agentic development you don’t always have to run and develop code on your machine. Sometimes you might want to outsource some tasks to remote locations. Enter Cloud Agents: asynchronous agents that can edit and run code and do not require explicit user control.

This risk here centers on an undocumented and potentially insecure default sharing mechanism for Cloud Agents.

By default, any user who shares the same GitHub repository-level access with another user is automatically granted read-only access to that user’s Cloud Agents created from that repository.

This sharing occurs silently and automatically, relying only on the external repository permission layer. This behavior is not documented in the Cloud Agents section and is hidden from the user who created the agent.

The Cursor UI provides a “Share” button intended for explicit agent sharing. The button triggers a popup that looks like this:

Our analysis confirmed that this button is non-functional on the backend. It merely takes the agent’s UUID and creates a full URL, and performs no authorization or permission changes on the server. The actual read-only access is entirely controlled by the implicit, default GitHub repository permissions.

This undocumented behavior can be leveraged by an attacker with co-worker access to facilitate unauthorized information gathering. Because the agent sharing is implicit, a user intending to keep their agents private, even if they explicitly avoid using the non-functional “Share” button, is unknowingly exposing their agent configuration and potentially its execution context to all collaborators with access to the source repository. This is a gap in the application’s security perimeter.

Proof of concept

By default, team members can’t list all the cloud agents started by other users in the same Cursor team. But we identified that endpoint https://cursor.com/api/background-composer/get-detailed-composer can be abused to disclose other users’ cloud agent instances.

The default request looks like:

POST /api/background-composer/get-detailed-composer HTTP/1.1

Host: cursor.com

[...]

{”bcId”:”bc-2404e33c-1d8d-4a29-b8f4-1da70b05fcbd”,”n”:1,”includeDiff”:true,”includeTeamWide”:true}

If we omit the bcid - a unique ID that identifies the cloud agent instance - however, the server will happily return the most recently-created cloud agent.

REQ:
POST /api/background-composer/get-detailed-composer HTTP/1.1
Host: cursor.com
[...]
{”n”:1,”includeDiff”:true,”includeTeamWide”:true}
RES:
HTTP/1.1 200 OK
[...]
Content-Type: application/json; charset=utf-8
Date: Wed, 19 Nov 2025 21:47:12 GMT
Server: Vercel
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Matched-Path: /api/background-composer/get-detailed-composer
X-Permitted-Cross-Domain-Policies: none
X-Vercel-Cache: MISS
X-Vercel-Id: arn1::iad1::2p9x8-1756890592479-dcf0a5acb845
X-Xss-Protection: 1; mode=block
[...]
{”composers”:[{”composer”:{”bcId”:”bc-7eb60f01-b9f6-42e1-ad24-4e3826b2d318”,”createdAtMs”:1763588776751,”workspaceRootPath”:”/workspace”,”isOwnedByDifferentTeamMember”:true,”name”:”Print custom environment variables”,”branchName”:”cursor/print-custom-environment-variables-7d95”,”hasStartedVm”:true,”repoUrl”:”github.com/TLBC-pl/Prompting”,”status”:”BACKGROUND_COMPOSER_STATUS_FINISHED”,”isUnread”:true,”source”:”BACKGROUND_COMPOSER_SOURCE_WEBSITE”,”updatedAtMs”:1763588788733,”modelDetails”:{”modelName”:”composer-1”,”maxMode”:true},”triggeredPrincipalType”:”user”,”triggeredPrincipalId”:”191726155”,”visibility”:”team”},”modelDetails”:{”modelName”:”composer-1”,”maxMode”:true}}]}

Now all the attacker needs to do is to access:

https://cursor.com/agents?selectedBcId=bc-26f0648e-5ec9-4ce6-b319-e346e83b5e21

Example attack

Victim side

• Create a user environment

• Start a Cloud Agent

• Perform some tasks

Attacker side

• Get the history

Impact

The vulnerability can be exploited by an insider—specifically, a Cursor team member in the same tenant who has access to the target GitHub repository-level access with the victim can exploit this flaw to enumerate the UUIDs of Cloud Agents created by other users. While write operations are correctly restricted, the read-only access enabled by the UUID leak results in the disclosure of the entire agent conversation, bypassing an implied privacy boundary for user-created assets.

For example, in a large company it might be the case that many different employees have GitHub repository-level access to an internal utility. By exploiting the described flaw in Cursor, an employee in the engineering department could review the entire cloud agent conversation of an employee in the finance department. This could lead to unintended sharing of sensitive data such as material nonpublic information.

We tried to follow-up on the conversation with the cloud agent, but could not. The example below shows our attempt of escalation.

The Cursor security team triaged our report and concluded this feature works as intended.

Recommendation

If using Cloud Agents, notify employees about the implicit sharing driven by GitHub repository access. Comprehensively and regularly review access to the GitHub repositories to ensure least privilege.

Issue #3: Chained authentication flow abuse leading to token replay and account takeover

Implementing a complex login process creates many opportunities for error. While analyzing that for Cursor, we did not find any glaring vulnerabilities. We did, however, find many small shortcomings that, when combined into a single attack scenario, could result in account takeover after minimal user interaction.

Specifically, we found a chained attack vector that combines multiple weaknesses, exploitation of which could result in account takeover (ATO) and installation of a persistent backdoor via user rules.

The attack is contingent upon successful social engineering of the victim.

Attack components

1. Malicious document injection: An attacker adds a specially crafted document containing malicious content.

2. Abuse of trust: The malicious content displays a seemingly legitimate, user-facing prompt or message, abusing the user’s trust in the application’s native UI.

3. Client-Side Execution: Interaction with the document executes code that, after a few user actions, initiates a sensitive workflow.

This lists the necessary user steps, emphasizing that the application’s intended function is being abused.

Required user actions for execution

The attacker must successfully induce the victim to perform the following steps:

• Add the malicious doc (Initial setup).

• Cause user to interact with the document (Triggering the payload).

• Trick user into clicking the “Yes, Log in” Button (Final confirmation to execute the sensitive action).

While this requires several user actions, the flaw lies in abusing an intended feature (the documents system) and the established trust with the user-facing messaging. The success of the attack hinges on the user trusting the deceptive prompt.

The multi-step account takeover vulnerability leverages a flawed token exchange mechanism and insufficient client-side user verification. The core login process, which we analyzed, operates as follows:

1. IDE Initialization: The user clicks “Log in” within the Cursor IDE chat panel, initiating a token polling process in the background.

2. Challenge generation: The IDE opens a browser window to a vendor-controlled page, presenting a login challenge.

3. Token setup: Upon the user completing the challenge by clicking “Yes, Log in,” the IDE pulls the resulting challenge information from the backend and establishes the authenticated session using the returned JWT (JSON Web Token).

Example of the login popup used on the Cursor page:

Aspect A: The replayable login link (IDE flaw)

The first aspect of the flaw resides in the IDE’s handling of the unique login link.

Problem: The IDE generates a unique login link for the session challenge, but this link remains accessible and usable even after the initial login flow has been completed.

Impact: This lack of single-use enforcement or link invalidation after fulfillment creates a window for abuse. An attacker who compromises this link could potentially replay the authentication flow or hijack a subsequent session.

Aspect B: Insufficient client-side verification (web UI flaw)

The second aspect of the flaw is the web-facing user experience, which is ripe for social engineering.

The prompt itself

The UI displays a message warning the user not to click “Yes, Log in” if the request originated from an untrusted source.

This message is ineffective in a social engineering attack because, in the victim’s eyes, Cursor is a trusted, legitimate application. The warning fails when the attacker successfully injects a malicious payload that appears to be part of the trusted application’s workflow.

Lack of context

The login page provides little contextual data about the authentication request (e.g., the initiating IP address, browser information, or the time the process was initiated).

The absence of crucial data—such as a timestamp that could flag a login process started hours ago—makes it harder for the user to detect a suspicious login attempt.

Aspect C: persistent token theft via auth polling (server-side flaw)

The final component of the attack chain leverages a server-side flaw in the authentication token exchange mechanism, specifically within the polling endpoint.

The IDE communicates with the backend via a dedicated polling endpoint:

https://api2.cursor.sh/auth/poll?uuid={uuidv4}&verifier={random_secret}

Upon successful authentication completion by the user, the server returns a HTTP 200 response containing the authenticated JWT, which the IDE then saves into its configuration.

The critical flaw: reusable challenge

The authentication challenge, identified by the unique uuid and verifier parameters, does not expire or invalidate after a successful login is performed.

This failure to enforce a single-use token challenge means the same challenge parameter remains valid indefinitely.

An attacker who successfully executes the social engineering trap on multiple users can then track the response body of the polling endpoint, stealing subsequent JWT tokens for every user who falls into the trap.

This transforms the vulnerability into a highly effective watering hole attack. The reusable nature of the challenge allows an attacker to repeatedly harvest new session tokens for compromised accounts.

Proof of concept

The vendor’s triage concluded that this vector was “working as intended,” arguing that the explicit user prompt (”Log in to Cursor Desktop”) functions as a “clear speedbump” intended to alert the user to suspicious activity.

They did not consider the kill-chain to be a vulnerability because of this required user action.

We disagree with this assessment of risk.

Our investigation confirmed that the initial malicious document injection successfully initiates the entire process from within the trusted Cursor environment. Because the user may perceive the request as originating from the legitimate application itself, the “speedbump” can be ineffective. The social engineering component can be successful precisely because the request does not appear to come from an untrusted source, undermining the vendor’s core rationale for discounting the severity of this persistent ATO chain.

Recommendation

To mitigate the risk inherent in highly integrated, chat-enabled platforms, organizations must prioritize employee awareness programs detailing how socially engineered prompt injections can be used to subvert system instructions and use otherwise legitimate interfaces to facilitate an attack.

Takeaways

When analyzing AI tools handling sensitive data, you can’t trust the vendor blindly. Technical verification of documentation and statements is key. This approach allows identifying and addressing security blind spots before they lead to an incident.

StackAware helps AI-powered companies measure and manage cybersecurity, compliance, and privacy risk. Importantly, we include all of your commercial and open source AI systems in our Relentless AI Red Teaming program, which led to the identification of these issues.

Are you managing AI coding assistants at a healthcare or enterprise software firm?

Need help?

Book a call

Appendix A - Timeline

• Aug 27, 2025 - Report of Issue #1 to Cursor via Github

• Aug 29, 2025 - Cursor confirms receipt

• Sep 01, 2025 - Report of Issue #2 to Cursor via GitHub

• Sep 03, 2025 - Report of Issue #3 to Cursor via GitHub

• Sep 15, 2025 - Reminder to Cursor about outstanding issues

• Sep 30, 2025 - Cursor completes triage of reported issues

• Sep 30, 2025 - Reporter provides comments on Cursor triage outcome

• Nov 20, 2025 - Reporter notification to the Cursor team about publication of this article, with opportunity to review and comment

• Nov 21, 2025 - Cursor team acknowledges planned publication

• Dec 01, 2025 - Publication of this article

I’m thrilled to announce that Eleos is ISO/IEC 42001:2023 certified!

A leading AI-powered platform for behavioral health, Eleos slashes administrative burdens and powers provider efficiency. But when handling sensitive protected health information (PHI), securing data and maintaining patient trust is absolutely critical.

Already ISO 27001 certified—and with a SOC 2 + HITRUST attestation—Eleos runs a tight privacy and security ship under Raz Karmi, Chief Information Security Officer. Due to the intricacies of AI—and the risks it poses if left ungoverned—he wanted to build a fully certified Artificial Intelligence Management System. He also needed to avoid common pitfalls while pursuing a relatively new compliance certification so he could focus on high-leverage strategic goals, not waste time on minutiae.

That’s where StackAware came in.

Assessing models, systems, risks, and impacts

Step 1 of our engagement was a thorough inventory of all assets in use. Because Eleos uses a range of model approaches and tech partners, understanding the entire risk landscape was critical. With that done, we then:

• Evaluated models for data quality, provenance, and undesired bias.

• Mapped ISO 42001 requirements to Eleos’ AI systems and products.

• For those processing PHI, we analyzed their societal and individual impacts.

• Logged risks where there were opportunities for improvement.

Building a solid foundation of governance

Using their existing Information Security Management System (ISMS) as a starting point, we then crafted actionable:

• AI-related policies and procedures

• Updates to existing compliance documentation

• Tech stack-specific governance training to educate employees

Raising the bar with an AI governance standard

With the right scaffolding in place, StackAware and Eleos could then map out the right AI-specific controls from ISO 42001’s Annex A. We also built an AI governance standard against which we could evaluate all systems and models to confirm they met company requirements and risk tolerances.

This involved a detailed look into these assets’:

• Intended use and risk profiles

• Observability, logging, and monitoring

• Data sensitivities and retention policies

“StackAware’s approach to ISO 42001 certification made things incredibly easy for me and the Eleos team. They focused on building a lean management system that addresses real threats to our AI operations, not just on creating compliance documentation. This let our team prioritize what matters most: delivering secure and effective AI-powered behavioral healthcare.”

- Raz Karmi, Chief Information Security Officer, Eleos

Managing risk and achieving compliance

At the end of the engagement, StackAware had delivered to Eleos:

• An effective and streamlined AIMS that could be easily integrated into business operations. Rather than a paperwork-heavy checkbox exercise, Eleos runs a lean management system focused on addressing risks and improving AI effectiveness.

• Clear and effective data and AI model governance. With realistic standards and procedures in place, fast-moving internal teams can develop and deploy AI in a repeatable and controlled manner.

• Proactive risk assessment and incident response. Through actionable recommendations, StackAware gave Eleos a clear roadmap to deal with any residual risk. At the same time, the company was prepared for the worst with an agile incident response procedure.

Driving value with AI governance

By partnering with StackAware, Eleos was able to achieve ISO 42001 certification without being distracted from its core mission: improving behavioral healthcare. As Raz noted, “Working with StackAware gets me out of the weeds so I can focus on our customers and guide our strategy.”

For us, success means enabling customers to build and scale innovative solutions while creating trust and avoiding breaches and potential legal penalties.

Are you a security, compliance, or technology leader in AI-powered healthcare looking to:

• Manage risk?

• Strengthen customer trust?

• Avoid regulatory scrutiny and fines?

StackAware delivers customized, white-glove ISO 42001 readiness. So please:

Book a call

AI mistakes don’t just cause chaos—they can leak data, corrupt systems, and crush a business.

A safeguard?

Human-in-the-loop (HITL).

Used wisely, it protects data confidentiality, integrity, and availability. But do it incorrectly, and you can slow innovation or even cause more damage than otherwise.

Here are three concrete ways to apply HITL:

1) Default deny

Nothing happens until a human approves. This is for life-or-death or heavily-regulated decisions like:

• Surgery

• Hiring decisions.

• Drone strikes (!).

Important note: laws like Colorado’s SB-205 and NYC Local Law 144 may still apply, even with default deny.

2) Default allow (with intervention window)

Actions run unless a human interrupts within a set time. Balancing speed with oversight, in this mode the system:

• Alerts someone before execution.

• Gives time to stop harmful steps.

• But proceeds by default.

This approach fits medium-risk use cases like:

• Database writes.

• Business emails.

• Public social media posts .

This is also generally how autonomous vehicles operate (if there is still a driver in the car). A human is observing the AI’s decisions but has the ability to override them.

3) Post-hoc review

Humans validate the system’s performance after the fact. By auditing the AI’s outputs against rules or manual benchmarks, humans can determine its performance.

This is good for:

• Fraud alerts.

• Marketing tests.

• Performance tuning.

This approach offers the lowest control but fastest throughput.

Post-hoc review can take the form of iterative human feedback to an AI system at conditions-based or scheduled points during its operation. In the most advanced case, the system would identify to its human operator the most ambiguous cases and ask for guidance to improve its performance over time.

Hybrid approaches

Confidence-sensitivity

An AI system could automatically choose which of the 3 approaches to use based on its confidence threshold. For example, in the cybersecurity context a zero trust policy engine could take the following actions based on how sure it was about the legitimacy of a given login attempt:

• <80% confidence: default deny (out-of-band notification to user).

• 80-90% confidence: default allow (with intervention from security team).

• >90% confidence: automatically approved (with post-hoc review).

Context-sensitivity

Similarly, a set of qualitative criteria could determine which approach is required. For example, in healthcare an AI might recommend medications using the below rule:

• Prescription: default deny (no recommendation made without human doctor).

• Over-the-counter: default allow (with intervention notification to human doctor).

Human-in-the-loop (HITL) can come in three forms

1. Default Deny

2. Default Allow

3. Post-Hoc Review

Importantly, the method you choose is itself a decision. And not making a call can have the same impacts as the wrong one.

Need help choosing the right tool for the job?

StackAware helps AI-powered companies build ISO 42001-compliant governance programs to:

• Manage risk.

• Build customer trust.

• Avoid costly fines and regulatory enforcement actions.

Ready to learn more?

Book a call

Thanks to Martin Koder for his comment recommending the paper “Humans in the Loop” by Rebecca Crootof, Margot E. Kaminski, and W. Nicholson Price II. Although I think the 3 approaches in my article entirely cover all forms of HITL, the paper did provide inspiration for some of the hybrid methods I described.

Apps like Zapier and n8n are powerful but risky. Here’s how to cope:

1) Excessive autonomy (workflows run past your intent)

What can go wrong:

• Chained AI functions ping customers by mistake.

• Bad prompts trigger database deletes or writes.

• Multi-provider loops spiral and amplify errors.

What you can do about it:

• Human-in-the-loop (HITL - native Zapier feature).

• Insert delays, budgets, and kill-switches.

• Gate risky actions behind approvals.

• Log every action. Alert on outliers.

• Dry-run before prod deployment.

2) Data confidentiality (accidental leaks / model training)

What can go wrong:

• A misrouted step exposes sensitive records.

• Vendors use your data to improve models.

• Chained tools multiply exposure paths.

What you can do about it:

• Review third party (e.g. OpenAI, Anthropic) terms.

• Use enterprise no-code tiers (no default training).

• Opt-out (like Zapier allows) when possible.

• Avoid built-in AI features if not.

• Redact, hash, or tokenize.

3) Compliance traps (residency, HIPAA, hiring laws)

What can go wrong:

• Data crosses regions, violates residency promises.

• Teams process PHI through tools without a BAA.

• Pre-built templates trigger AI-specific laws like:

  • NYC Local Law 144

  • Colorado's SB-205

  • California ADS reg

What you can do about it:

• Train no-coders on key compliance obligations.

• Check data processing addenda and subprocessor lists.

• Require legal review of any workflows touching human resources.

The StackAware no-code AI governance playbook

• Constrain: App allowlist (Zapier only) + least privilege.

• Contract: Lock training opt-outs and data residency.

• Decide: Which steps require human approval?

• Design: HITL gates, budgets, and kill-switches.

• Train: Teach builders the rules & legal triggers.

• Prove: keep logs, bias audits, and impact reviews.

• Test: Red-team failure modes before go-live.

• Strip: Send minimum data needed.

Bottom line

Treat no-code automation as a production system: with audits, approvals, and contracts.

Or expect painful surprises.

Need help securing your no-code / AI stack?

Book a call

Related LinkedIn Post:

• How to limit the blast radius from no-code tools

"Heads-up: P1 Security vulnerability on StackAware API"

👆 subject line of an email I got on August 28th.

This email, allegedly from "Sean Whitaker" at SecureForge, got my attention.

And raised suspicions.

Subscribe now

"Sean" asked how "to report an account takeover vulnerability I have discovered in StackAware API."

Two things were strange:

1. I've never heard of an account takeover for an API.

2. StackAware has a public security.txt file.

An experienced ethical hacker would have gone there and found our vulnerability disclosure program (VDP).

Wanting to act in good faith in case he was legit, I directed him to our VDP.

6 minutes later, he replied, making my spidey senses tingle even more: "Quick question, do you offer any form of compensation or bug bounty rewards for high impact security vulnerabilities like this one?"

The obvious answer, if you read our VDP, is no.

We love ethical hackers and work with many good ones, but StackAware is a bootstrapped startup and cannot afford bug bounties (yet).

The fact he ignored our VDP and quickly asked about compensation raised another red flag.

So I started investigating.

If you go to secureforge [dot] io1 (archive.org version here, for when they inevitably get taken down), at first glance it 𝘭𝘰𝘰𝘬𝘴 kind of legitimate. But dig deeper and find:

• They claim to be trusted by "500+ enterprises"

• The only way to contact is a complex form

• None of the buttons on the website work

They also claimed to be "Securing Digital Infrastructure Since 2015," but their site was registered in July 2025.

They even have an FAQ page that says:

"SecureForge is a registered security firm with verifiable credentials. You can verify our legitimacy through our CVE contributions, OWASP membership, client testimonials, and professional certifications."

• "SecureForge OWASP" has no relevant Google hits.

• Neither does the query "SecureForge CVE."

• Testimonials are from real people, but...

...I saw a LinkedIn post by one of them, Nacho García Egea, claiming he never made the endorsement!2

That put the nail in the coffin.

I don't know what "Sean" planned next, but expect it would be something along the lines of "pay us or we won't give you the [non-existent] vulnerability."

So I used the "Report phishing" function on Gmail and reported the domain to Google.

This is unfortunately a fairly common approach, whereby unethical hackers attempt to "gray mail" companies into paying up.

Which makes clear disclosure policies key - they let you identify bad actors quickly.

Recommendations

• Verify everything, especially if it seems urgent

• Have a vulnerability disclosure program (VDP)

• Be clear whether you pay (or not!)

  Subscribe now

California Civil Rights Department regulatory action 2025-0515-01 on Automated-Decision Systems in employment goes into effect October 1, 2025.

Subscribe now

The key section is § 11009 (f):

It is unlawful for an Employer or Other Covered Entity or other covered entity to use an Automated-Decision System or selection criteria (including a qualification standard, employment test, or Proxy) that discriminates against an Applicant or Employee or a class of Applicants or Employee on a basis protected by the Act, subject to any available defense.

Most importantly:

Relevant to any such claim or available defense is evidence, or the lack of evidence, of anti-bias testing or similar proactive efforts to avoid unlawful discrimination, including the quality, efficacy, recency, and scope of such effort, the results of such testing or other effort, and the response to the results.

But what is "anti-bias testing"?

Not defined.

But StackAware has some suggestions, which we document in an actionable compliance procedure (below).

The regulation also does not mention ISO 42001 or similar standards, but external certification of an AI Management System (with relevant Annex A controls) could address the criteria of:

• Quality

• Efficacy

• Response

And the ADS regulation could be an “external issue” which Clause 4 of the ISO 42001 standard requires you to consider.

Notes on the procedure:

• It assumes you are an “Employer” per the regulation (which has a complex definition but basically covers all businesses with >5 employees if any of them are in California).

• It is not a comprehensive human resources policy that covers all anti-discrimination requirements, nor is it legal advice.

• Capitalized terms are defined in the California regulation.

Purpose

Ensure the responsible and ethical use of Automated-Decision Systems as well as compliance with California Civil Rights Department regulatory action 2025-0515-01.

Scope

All information systems1 COMPANY_NAME develops2 or uses that impact (or could in the future impact) employees or legal equivalents located in California.3

Requirements

Data Owners must:

• NOT use Automated-Decision Systems to discriminate against an employee or legal equivalent on the basis of (or proxy for) any legally protected class4, or attempt to identify an employee or legal equivalent on such a basis (§ 11009 (f)).

• Annually, conduct bias mitigation5 of any Automated-Decision Systems, including one of the following or alternative approved by the General Counsel:

  • A "Bias Audit" as defined by NYC Local Law 1446

  • A/B testing of outputs from the ADS

  • Review of ADS training data

  • AI Red-teaming

• Retain, for 4 years from the date of the making of the record or the date of the personnel action involved, whichever is later all (§ 11013 (c)):

  • applications

  • personnel records

  • membership records

  • employment referral records

  • selection criteria

  • Automated-Decision System Data

  • California Employer Information Reports (CEIR)

  • Applicant Identification Records

  • other records created or received by the Employer or Other Covered Entity or other covered entity dealing with any Employment Practice and affecting any Employment Benefit of any Applicant or Employee.

• Keep records as to the sex, race, or national origin of any individual accepted for employment separate from the employee's main personnel file or other records available to those responsible for personnel decisions (§ 11013 (c)(2)).

• If an online application technology7 limits, screens out, ranks, or prioritizes Applicants based on:

  • schedule or time availability (§ 11016 (c)(3))

  • skill, dexterity, reaction time, and/or other abilities or characteristics

    (§ 11016 (c)(5))

  • tone of voice, facial expressions, or other physical characteristics or behavior (§ 11016 (d)(1))

  • then:

    • Document and retain for 4 years the business necessity and job-related nature of the schedule or availability restriction or preference; and

    • Ensure the online application technology offers a method for the applicant to request an accommodation.

Subscribe now

AI is reshaping healthcare as we know it. Companies using artificial intelligence effectively and responsibly will lead the industry and change patient lives for the better.

But in this high-stakes field, this demands trust.

Rightway's AI leadership

Rightway is at the forefront of this change. A healthcare company helping people find the doctors and medicines they need, Rightway combines smart technology with human guidance to facilitate care navigation and pharmacy benefits management (PBM).

While maintaining a human touch and ensuring its members feel they have a “doctor in the family,” the company is simultaneously deploying AI to accelerate employee productivity and let them make better decisions.

To do so responsibly, though, the company had three primary concerns:

Cybersecurity

In addition to regulated Protected Health Information (PHI), Rightway needed to protect trade secrets like its pricing strategies and proprietary approaches to benefits delivery.

Stopping data leaks stemming from AI use was critical to reduce the risk of costly penalties, reputation damage, and lost competitive advantage.

Compliance

On top of the Health Insurance Portability and Accountability Act (HIPAA), which mandates PHI security and privacy measures, Rightway had to contend with a variety of other compliance obligations. For example, it already maintains a System and Organization Controls (SOC) 2 + HITRUST attestation.

Trust

Because it serves large enterprises as customers, Rightway had to demonstrate responsible AI use to the savvy security teams evaluating it. Complex contractual requirements could hamstring the business, so Rightway took the initiative to build its own AI governance framework as a starting point.

Rightway members also rely on the company to keep their sensitive health data private.

Building an AI Management System with StackAware

To tackle the three key challenges it faced, Rightway contracted with StackAware to custom-build an ISO/IEC 42001:2023-compliant AI Management System (AIMS). We provided Rightway with:

• Comprehensive regulatory and business issues analysis

• Detailed risk and impact assessments

• Tailored policies and procedures

and everything it needed to build and maintain an effective AIMS. Rightway also opted for a continuous monitoring and governance support retainer to avoid compliance drift and manage risk effectively over time.

“While the human touch will always remain, AI lets us guide members more effectively and efficiently. As we deployed artificial intelligence throughout the company, I was deeply concerned about data protection, privacy, and regulatory compliance. StackAware helped Rightway address all of these issues, and more. Our AI is secure, follows the rules, and truly builds patient trust.”

— Jason Melo, Head of Information Security & Technology, Rightway

Leading with confidence

With an effective AIMS in place, Rightway can:

• Confidently adapt to and comply with new AI rules, cutting legal/regulatory risk.

• Communicate clearly and openly with customers to boost trust.

• Track performance over time and continually improve.

• Identify and manage AI-specific risks.

• Make better business decisions.

Building trust and speeding AI adoption in healthcare

Are you a security, technology, or business leader at an AI-powered healthcare company that needs to:

• Control risk?

• Stay compliant?

• Build patient trust?

The StackAware team can help.

Book a call

The artificial intelligence (AI) attack surface is expanding—fast.

Models can:

• Leak data

• Misbehave under pressure

• Create new attack surfaces

StackAware’s answer: Relentless AI Red Teaming.

It’s not a scan.

It’s not a one-time audit.

It’s a continuous, full-coverage assault on your AI systems.

How does it work?

• You keep us up-to-date on your models and systems.

• We give an anonymized list to ethical hackers.

• They scour it to find vulnerabilities.

• We tell vendors to patch the bugs.

• Or help mitigate risk if they don’t.

What do I get?

1. Continuous assessment: this isn’t an annual review. We test vendors and models you use continuously for both AI-specific and infrastructure-level vulnerabilities.

2. Ethical by design: we follow vendor disclosure, bug bounty, and terms and conditions. Our goal is to expose risks, not generate drama.

3. You’re always covered: if a vendor ignores a bug, we help you apply compensating controls. We don’t disclose anything publicly unless there is a fix in place. And you’ll get insights on how to manage the risk before a patch is available.

Pursuing or maintaining ISO 42001 certification?

This helps implement several annex A controls like:

• A.6.2.4 - AI system verification and validation

• A.6.2.6 - AI system operation and monitoring

• A.8.3 - External reporting

Who does the testing?

StackAware is proud to partner with Daniel Kalinowski and team as part of the pilot program. With 322 (and counting) vulnerabilities identified and responsibly disclosed, he is a leader in ethical hacking and AI system verification.

How do I sign up?

This is LIVE for current StackAware customers. And a feature all new ones get going forward.

So if you need help managing AI risk, and are a security or technical leader in:

• Life sciences

• Healthcare

• B2B SaaS

  Book a call

With the collapse of efforts to ban state-level AI regulation until 2035, the Colorado Artificial Intelligence Act (Senate Bill 24-205 or SB-205) will take effect on February 1, 2026.

As I wrote previously, SB-205 will have a massive impact on the American AI governance landscape. So to complement the Deployer compliance procedure I put together, I am now releasing a Developer version of it.

This procedure presumes:

• Your company is only a “Developer” of a “High-risk Artificial Intelligence System.” If your company is also a “Deployer,” incorporate this procedure as well. Note that Data Owners are not required to create the documentation described below in this case [6-1-1702 (3) (b)].

• You already have an ISO 42001-compliant or NIST AI RMF-adherent risk management policy and program. Interestingly enough, this is not required for Developers (like it is for Deployers), but having such a program is an affirmative defense against allegations of some violations under certain conditions [6-1-1706 (3)]. So it’s probably a good idea to have one.

• Your company doesn’t qualify for any of the (very complex) carve-outs in the law.

• Attempting to identify and selectively apply SB-205’s requirements to Colorado “Consumers” (residents of Colorado) per the law is too difficult, and you will just apply it to all of your customers (except any notification requirements to the Colorado Attorney General).

• It is February 1, 2026 or later.

Finally:

• References to specific sections of the law are in brackets ([]).

• Capitalized terms are defined in SB-205.

• This is not legal advice.

Purpose

Ensure the responsible and ethical development of Artificial Intelligence Systems as well as compliance with the Colorado Artificial Intelligence Act (SB-205).

Scope

All Artificial Intelligence Systems COMPANY_NAME develops that impact (or could in the future impact) residents of the State of Colorado.

Requirements

The Chief Information Security Officer must:

• Affirmatively document whether a given System is a High-Risk Artificial Intelligence System per SB-205.

• If so, ensure adversarial testing or red teaming of the System at least annually [6-1-1706 (3) (a) (II)].1

Data owners of High-risk Artificial Intelligence Systems must:

• Use reasonable care—including ensuring a review of the system prior to use, and at least annually thereafter [6-1-1702 (1)]—to ensure that the System does not cause Algorithmic Discrimination [6-1-1703].

• Ensure any System with which an end user interacts discloses it is an Artificial Intelligence System [6-1-1704 (1)].2

• Prior to allowing any third-party Deployer access to the System, provide the below to the Deployer and confirm receipt in writing [6-1-1702 (2)]:

  • System purpose [6-1-1702 (2) (b) (III)].

  • Intended System use [6-1-1702 (2) (c) (V)].

  • Prohibited System uses for Deployers [6-1-1702 (2) (c) (V)].

  • Intended benefits and uses of the System [6-1-1702 (2) (b) (IV)].

  • Intended System outputs [6-1-1702 (2) (c) (III)] and how to monitor them [6-1-1702 (2) (d)].

  • Reasonably foreseeable uses and known harmful or inappropriate System uses [6-1-1702 (2) (a)].

  • Known and reasonably foreseeable limitations of the System [6-1-1702 (2) (b) (II)].

  • Known or reasonably foreseeable risks of Algorithmic Discrimination arising from intended System use [6-1-1702 (2) (b) (II)].

  • Measures taken to mitigate such risks [6-1-1702 (2) (c) (IV)].

  • Methods to monitor such risks [6-1-1702 (2) (d)].

  • How the System was evaluated for mitigation of Algorithmic Discrimination [6-1-1702 (2) (c) (I)].

  • Recommend human oversight measures when making, or being a Substantial Factor in, a Consequential Decision [6-1-1702 (2) (c) (V)].

  • Summaries of the type of data used to train the System [6-1-1702 (2) (b) (I)].

  • How the System was evaluated for performance [6-1-1702 (2) (c) (I)].

  • Data governance measures for training data including [6-1-1702 (2) (c) (II)]:

    • Measures used to examine the suitability.

    • Possible biases.

    • Appropriate mitigation(s).

  • Any other information required for the Deployer to comply with the StackAware Colorado SB-205 Deployer procedure [6-1-1702 (2) (b) (V) and 6-1-1702 (3) (a)].

• Prior to System deployment [6-1-1702 (4) (a)], Intentional and Substantial Modification [6-1-1702 (4) (b) (II)], and otherwise at least annually, review COMPANY_NAME web site and ensure the following are available:

  • System name and type [6-1-1702 (4) (a) (I) ]

  • Methods for managing known or reasonably foreseeable risks of Algorithmic Discrimination arising from System use [6-1-1702 (4) (a) (II)].

The General Counsel must:

• Within 30 days of discovery of a known or reasonably foreseeable risk of Algorithmic Discrimination arising from the intended use of the System impacting a Colorado resident, notify [6-1-1702 (5)]:

  • All known Deployers of the System

  • All other Developers of the System

  • The Attorney General of Colorado in the manner that office prescribes.

All employees and contractors must:

• Report, within 24 hours, to the General Counsel any known, suspected, or imminent occurrences of Algorithmic Discrimination.

Despite the efforts of many, the Colorado Artificial Intelligence Act (Senate Bill 24-205 or SB-205) is set to take effect on February 1, 2026.

I don’t think people yet understand, however, how big an impact this law will make on the American AI governance landscape.

Since you are probably reading Deploy Securely to get actionable information, I won’t get into the policy or politics. Rather, I’m happy to share this actionable SB-205 compliance procedure. It takes the laws requirements and consolidates them as much as possible into something you can actually use.

This procedure presumes:

• Your company is a “Deployer” of a “High-risk Artificial Intelligence System.” “Developers” have an additional set of requirements, which I tackle in this post.

• You already have an ISO 42001-compliant or NIST AI RMF-adherent risk management policy and program “specify[ing] and incorporat[ing] the principles, processes, and personnel that the Deployer uses to identify, document, and mitigate known or reasonably foreseeable risks of Algorithmic Discrimination” as required by Section 6-1-1703 (2) (a) of the law.

• Your company doesn’t qualify for any of the (very complex and hard to understand) carve-outs in the law.

• Attempting to identify and selectively apply SB-205’s requirements to Colorado “Consumers” (residents of Colorado) per the law is too difficult, and you will just apply it to all of your customers (except any notification requirements to the Colorado Attorney General).

• It is February 1, 2026 or later.

Finally:

• References to specific sections of the law are in brackets ([]).

• Capitalized terms are defined in SB-205.

• This is not legal advice.

Purpose

Ensure the responsible and ethical use of artificial intelligence as well as compliance with the Colorado Artificial Intelligence Act (SB-205).

Scope

All Artificial Intelligence Systems COMPANY_NAME operates–whether developed internally or procured from a third party–that impact (or could in the future impact) residents of the State of Colorado.

Requirements

The Chief Information Security Officer must:

• Affirmatively document whether a given System is a High-Risk Artificial Intelligence System per SB-205.

• If so, ensure adversarial testing or red teaming of the System at least annually [6-1-1706 (3) (a) (II)].1

Data owners of High-risk Artificial Intelligence Systems must:

• Use reasonable care—including ensuring a review of the system prior to use, and at least annually thereafter [6-1-1703 (3) (g)]—to ensure that the System does not cause Algorithmic Discrimination [6-1-1703].

• Request from the System Developer information per this questionnaire, unless COMPANY_NAME is itself the Developer [6-1-1702].

• Affirmatively notify any user of the System that it is an Artificial Intelligence System [6-1-1704 (1)].2

• Prior to use, within 30 days of System’s Intentional and Substantial Modification, and at least annually thereafter, ensure documentation of an impact assessment for the System covering its:

  • Purpose [6-1-1703 (3) (b) (I)].

  • Intended use case(s) [6-1-1703 (3) (b) (I)].

    • Following Intentional and Substantial Modification, a statement how this is consistent with, or varied from, the Developer's intended use(s) [6-1-1703 (3)(c)].

  • Deployment context [6-1-1703 (3) (b) (I)].

  • Benefits afforded [6-1-1703 (3) (b) (I)].

  • Known or reasonably foreseeable risk(s) of Algorithmic Discrimination [6-1-1703 (b) (II)].

    • If so, the nature of the Algorithmic Discrimination and the steps taken to mitigate the risk(s) [6-1-1703 (3) (b) (II)].

  • Categories of data:

    • Inputs to the System [6-1-1703 (3) (b) (III)].

    • Outputs produced by the System [6-1-1703 (3) (b) (III)].

    • Used in prompt engineering, retrieval-augmented generation (RAG), fine-tuning or any other process to customize the System [6-1-1703 (b) (IV)].

  • Metrics used to evaluate performance [6-1-1703 (b) (V)].

  • Known limitations [6-1-1703 (b) (V)].

  • Transparency measures, including measures to disclose use of the System [6-1-1703 (3) (b) (VI)].

  • Post-deployment monitoring and user safeguards, including processes related to [6-1-1703 (3) (b) (VII)]:

    • Oversight.

    • Use.

    • Learning process and continual improvement.

• Retain impact assessments for 3 years following final System deployment [6-1-1703 (3) (f)].

• Ensure the following are posted publicly on the COMPANY_NAME web site and reviewed within 30 days of the System’s Intentional and Substantial Modification, and at least annually thereafter:

  • Types of High-risk Artificial Intelligence Systems deployed [6-1-1703 (5) (a) (I)].

  • Description for each [6-1-1703 (4) (a) (II)].

  • Purpose of each System [6-1-1703 (4) (a) (II)].

  • Nature of Consequential Decision(s) made by each [6-1-1703 (4) (a) (II)].

  • Nature, source, and extent of the information collected and used by COMPANY_NAME, preferably via hyperlink to COMPANY_NAME’s privacy policy [6-1-1703 (5) (a) (III)].

  • COMPANY_NAME contact information [6-1-1703 (4) (a) (II)].

  • Instructions to opt out of the processing of personal data “for purposes of profiling in furtherance of decisions that produce legal or similarly significant effects concerning the Consumer” per the Colorado Privacy Act, section 6-1-1306 (1)(a)(I)(C) [6-1-1703 (4) (a) (III)].3

  • How COMPANY_NAME manages known or reasonably foreseeable risks of Algorithmic Discrimination [6-1-1703 (5) (a) (II)].

• As part of any communications regarding a Consequential Decision, notify the customer that COMPANY_NAME is using System to make, or be a Substantial Factor in making, the Consequential Decision. Include a hyperlink to the above website [6-1-1703 (4) (c) (I)(A)].

• If the Consequential Decision is adverse4 to the customer, also provide [6-1-1703 (4) (b)]

  • A statement disclosing the principal reason(s) for the Consequential Decision [6-1-1703 (4) (b) (I)], including:

    • How and to what degree the System contributed to the Consequential Decision [6-1-1703 (4) (b) (I)(A)].

    • Data processed by the System in making the Consequential Decision [6-1-1703 (4) (b) (I)(B)].

    • Source(s) of this data [6-1-1703 (4) (b) (I)(C)].

  • An opportunity to correct any incorrect Personal Data processed as part of the Consequential Decision [6-1-1703 (4) (b) (II)].

  • An opportunity to appeal an adverse Consequential Decision. If technically feasible, allow for human review unless providing the opportunity for appeal is not in the best interest of the customer [6-1-1703 (4) (b) (III)].

• Ensure all notifications to customers are:

  • In plain language [6-1-1703 (4) (c) (I)(B)].

  • In all languages in which COMPANY_NAME provides information to customers [6-1-1703 (4) (c) (I)(C)].

  • In a format that is accessible to customers with disabilities [6-1-1703 (4) (c) (I)(D)].

The General Counsel must:

• Within 30 days5 of discovery of an instance of Algorithmic Discrimination impacting a Colorado resident, notify the Attorney General of Colorado in the manner that office prescribes [6-1-1703 (7)].

All employees and contractors must:

• Report, within 24 hours6, to the General Counsel any known, suspected, or imminent occurrences of Algorithmic Discrimination.

One of the less talked-about AI regulations is the Illinois AI Video Interview Act (AIVIA, Public Act 101-260).

While the law is relatively short, it leaves many terms undefined (including “artificial intelligence” and “artificial intelligence program”). While there does not appear to have been much enforcement action related to it, I expect this to change as AI-related employment issues become a hotter topic.

So below is an actionable procedure to help comply with the law’s requirements.

As always, it is not legal advice.

Need help navigating the complex (and changing) landscape of AI regulation?

Book a call

Purpose

To ensure equitable hiring practices as well as compliance with the Illinois AI Video Interview Act (AIVIA, Public Act 101-260).

Scope

Artificial Intelligence Systems used to analyze recorded video interviews of employment applicants for positions in the State of Illinois.

Requirements

Data owners must:

• Not solely rely1 upon artificial intelligence analysis of a video interview to determine whether an applicant will be selected for an in-person interview and document additional criteria used for such selection.

• Provide to each applicant, either via email or through the Artificial Intelligence System itself2, before the interview:

  • Notification artificial intelligence may be used to analyze the applicant's video interview and consider the applicant's fitness for the position.

  • An explanation of how the Artificial Intelligence System works and what general types of characteristics it uses to evaluate applicants.

• Ensure (and document completion of the requirement) either:

  • The candidate consents, before the interview and via email or through the Artificial Intelligence System itself, to be evaluated by artificial intelligence; or

  • The candidate’s recorded video interview is not evaluated using artificial intelligence.

• Ensure recorded videos are only shared with persons whose expertise or technology is necessary in order to evaluate an applicant's fitness for a position.

• Ensure any third parties with access to recorded video interviews of applicants analyzed with Artificial Intelligence Systems are contractually obligated to delete all copies upon request.

• Ensure retention (and then destruction) of the following records per below:

  • Notification artificial intelligence may be used | 3 years from provision.

  • Explanation of how the Artificial Intelligence System works and what general types of characteristics it uses to evaluate applicants | 3 years from provision.

  • Candidate’s consent to be evaluated by artificial intelligence | 3 years from provision

  • All copies of the applicant's interview video | 30 days after applicant requests they be destroyed.

For complex manufacturing, infrastructure, and aerospace companies, AI-powered transformation presents a monumental opportunity. Engineering and R&D organizations that leverage artificial intelligence to accelerate innovation will define the future.

But this opportunity requires a foundation of trust. When AI is part of critical engineering workflows and product design, the stakes are incredibly high. Companies that proactively build robust governance around their AI systems not only mitigate risk, they earn the confidence of customers, accelerate sales, and lead their industries.

Those that don’t risk being left behind.

Accuris as an AI-powered leader

As the leading platform for digital engineering, Accuris is at the forefront of this transformation. Their suite of products, including:

• Parts & Bill of Materials (BOM) Intelligence

• Engineering Workbench

• Accuris Thread

• Goldfire

Accuris empowers the world’s most innovative companies to consolidate content from over 450 standards partners and digitize manufacturing requirements.

Understanding that leadership requires not just innovation but also responsibility, Accuris became an early adopter of a comprehensive AI governance strategy, ensuring their AI-powered tools were not only powerful but also secure, compliant, and trustworthy.

“AI is radically improving engineering workflows and productivity across industries. But as a security leader, I was deeply concerned about the accompanying risks to data confidentiality, integrity, and availability. Walter and the StackAware team helped Accuris lead the way when it comes to secure, compliant, and privacy-preserving AI.”

- Tim Volckmann, Chief Information Security Officer, Accuris

At the same time, Accuris’s leadership team had a thorough appreciation of the challenges that came along with this new technology.

Cybersecurity

Accuris’s platforms are the lifeblood of customer R&D efforts, containing sensitive intellectual property, proprietary designs, and critical parts intelligence. Protecting this data from leakage or misuse via AI systems was a top priority. Understanding and mitigating the risk of unintended training and the potential for AI to generate inaccurate outputs were also key concerns.

Compliance

With a global customer base, Accuris must navigate a complex and evolving landscape of international regulations. New requirements like the European Union (EU) AI Act and the Colorado Artificial Intelligence Act (SB24-205) are on the horizon. So the firm needed a structured, repeatable way to ensure and demonstrate compliance, avoiding potential fines and market access restrictions.

Trust

For any enterprise platform, customer confidence is essential. For one that handles critical engineering data, it is the entire business. Accuris recognized that simply having strong security controls wasn’t enough. They needed to demonstrate their commitment to responsible AI. Proving they had a robust AI Management System (AIMS) in place was crucial for strengthening customer relationships and shortening enterprise sales cycles.

“Protecting our customers' data and their trust in our platform is non-negotiable. The explosion in AI capabilities creates immense value, but also new risks. The StackAware team did an incredibly thorough job building our AI governance program. The systems they built for us manage the full range of AI-related risks and give our customers peace of mind.”

- Trenton Pologar, Chief Information Officer, Accuris

Enter StackAware

Given its stringent requirements for world-class AI governance, Accuris contracted with StackAware to build a comprehensive AI Management System and achieve ISO/IEC 42001:2023 readiness. StackAware quickly delivered by:

Building a foundation of trust with a customized AIMS

StackAware created a comprehensive set of controls to address identified risks. Using the National Institute of Standards and Technology (NIST) AI Risk Management Framework (RMF) and ISO 42001 standard as guides, StackAware delivered a suite of custom policies and procedures that wove AI governance directly into the firm’s existing security program. This provided a crucial benefit: instead of a theoretical checklist, Accuris got a practical, operational AIMS that empowered their teams to innovate safely.

Enabling confident decision-making

StackAware implemented a series of registers and assessment procedures that gave Accuris a clear, continuous view of its AI landscape. By creating a process to evaluate data provenance, quality, and bias for the company’s AI models, Accuris moved from reactive to proactive in terms of its risk management posture. This approach lets Accuris to confidently deploy new AI features while knowing risks are identified, measured, and mitigated according to a clear, defensible standard.

Empowering teams to innovate securely

While effective governance systems are powerful, only by educating employees can a company unlock the full potential of AI. StackAware delivered detailed training modules to help Accuris employees understand best practices for secure and responsible AI use. This investment in people ensures that the principles of the AIMS are put into practice daily, creating a culture of responsible innovation across the entire organization.

Ready for ISO 42001 and beyond

With a comprehensive AI Management System in place, Accuris is perfectly positioned to lead in the digital engineering space. The engagement with StackAware delivered tangible business value by enabling Accuris to:

• Accelerate sales: By adopting ISO 42001, Accuris can proactively demonstrate its commitment to security, satisfy enterprise procurement requirements faster, and build immediate trust with prospects.

• Strengthen customer trust: The AIMS provides verifiable proof of Accuris's commitment to protecting customer data and using AI responsibly, reinforcing its brand as a trusted, ethical partner.

• Manage global regulatory risk: With a governance framework aligned to global standards, Accuris is prepared to meet the demands of emerging AI regulations. This protects the business from fines and reputation damage.

With the right AI governance in place, Accuris can focus on what it does best: building the future of digital engineering.

Ready to build trust and accelerate sales with responsible AI?

If you are a security, compliance, or product leader at an AI-powered company and need to empower your teams to use AI while:

• Managing risk

• Staying compliant

• Keeping customer trust

the StackAware team can be your full service partner.

Interested in learning about how we can work together to build actionable and compliant AI governance programs?

Book a call

The web of AI regulation is thick, and growing.

New York City led the pack by passing its own Local Law 144 in 2021, which regulates the use of Automated Employment Decision Tools (AEDT).

I have reviewed several AI-powered recruiting tools that are almost certainly being used as AEDTs, and only one of them fully complies with this law…

…by entirely disabling its AI functionality for candidates in New York City.

Business demands will eventually make this approach infeasible, though, so I put together a simplified procedure for compliance that take into account the law itself and the enforcing regulation that greatly expands on some of the definitions.

This isn’t legal advice, so check with your attorneys when integrating this into your AI governance program.

But it gives you an actionable starting point.

New York City Local Law 144 Compliance Procedure

Purpose

To ensure equitable hiring practices as well as compliance with New York City’s Local Law 144.

Scope

Artificial Intelligence systems1 used for Employment Decisions within the 5 boroughs of New York City.

Requirements

Data owners must ensure:

• The system is not an Automated Employment Decision Tool (AEDT) because it does not provide Simplified Output related to Employment Decisions (and affirmative documentation of this fact);

• The system is not an AEDT because, for all Employment Decisions, the system’s Simplified Output is not weighted more than any other criteria2 (and documented support of this assessment for each Employment Decision); or

• If the system is an AEDT:

  • The following are available on COMPANY_NAME web site:

    • Hyperlink to a summary of results of a Bias Audit of the AEDT completed in the last year, including at least:

      • Date conducted.

      • Source and explanation of data used.

      • Number of applicants or candidates.

      • Selection or scoring rates.

      • Impact ratios for all categories.

      • Number of individuals the system assessed that fall within an unknown category (not identified by sex and race/ethnicity).

    • Type(s) and source(s) of data collected by AEDT.

    • Retention policy for such data.

    • Date of first use of AEDT.

  • Candidates or employees are notified via email–or the AEDT itself–15 or more days3 prior to assessing them with an AEDT:

    • That COMPANY_NAME will use an AEDT to do so.

    • About which job qualifications the AEDT assesses.

    • How to opt-out of assessment by the AEDT and request an alternative selection process or accommodation.

  • Retention of the above notification for 3 years.

• Candidates who request an alternative selection process or accommodation—not using an AEDT—are afforded one.4

Definitions

“Artificial Intelligence (AI) system,” means an engineered system that generates outputs such as content, forecasts, recommendations[,] or decisions for a given set of human-defined objectives. (per ISO/IEC 22989:2022).

“Automated Employment Decision Tool” means any computational process, derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues simplified output, including a score, classification, or recommendation, that is used to substantially assist or replace discretionary decision making for making employment decisions that impact natural persons. The term "automated employment decision tool" does not include a tool that does not automate, support, substantially assist or replace discretionary decision-making processes and that does not materially impact natural persons, including, but not limited to, a junk email filter, firewall, antivirus software, calculator, spreadsheet, database, data set, or other compilation of data.

“Bias Audit” means an impartial evaluation by an independent auditor. Such a bias audit shall include but not be limited to the testing of an automated employment decision tool to assess the tool’s disparate impact on persons of any component 1 category required to be reported by employers pursuant to subsection (c) of section 2000e-8 of title 42 of the United States Code as specified in part 1602.7 of title 29 of the Code of Federal Regulations.

New York City provides granular requirements for a Bias Audit in the enforcing regulation of Local Law 144.

“Distribution Date” means the date the employer or employment agency began using a specific AEDT.

“Employment Decision” means to screen candidates for employment or employees for promotion within the city.

“Impact Ratio” means either (1) the selection rate for a category divided by the selection rate of the most selected category or (2) the scoring rate for a category divided by the scoring rate for the highest scoring category.

“Machine learning, statistical modeling, data analytics, or artificial intelligence” means a group of mathematical, computer-based techniques:

i. that generate a prediction, meaning an expected outcome for an observation, such as an assessment of a candidate's fit or likelihood of success, or that generate a classification, meaning an assignment of an observation to a group, such as categorizations based on skill sets or aptitude; and

ii. for which a computer at least in part identifies the inputs, the relative importance placed on those inputs, and, if applicable, other parameters for the models in order to improve the accuracy of the prediction or classification.

“Screen” means to make a determination about whether a candidate for employment or employee being considered for promotion should be selected or advanced in the hiring or promotion process.

“Simplified Output” means a prediction or classification as specified in the definition for "machine learning, statistical modelling, data analytics, or artificial intelligence." A simplified output may take the form of a score (e.g., rating a candidate's estimated technical skills), tag or categorization (e.g., categorizing a candidate's resume based on key words, assigning a skill or trait to a candidate), recommendation (e.g., whether a candidate should be given an interview), or ranking (e.g., arranging a list of candidates based on how well their cover letters match the job description). It does not refer to the output from analytical tools that translate or transcribe existing text, e.g., convert a resume from a PDF or transcribe a video or audio interview.

Artificial Intelligence (AI) presents a massive opportunity for companies to increase velocity, productivity, efficiency, and innovation. However, given its fast-changing nature and employee demand, risks arise when businesses and customers outpace the ability to ensure scalable, secure, and private AI services

These risks can include:

• Data leakage

• Ungoverned use

• Regulatory non-compliance

• Unexpected costs

• AI ecosystem changes

• Complexity and instability

To navigate this complex terrain, LastPass partnered with StackAware, a firm specializing in AI risk management and governance.

Challenge

As LastPass carefully integrates AI across its products and operations, our focus is to ensure responsible, ethical, safe and secure use while maintaining a stringent emphasis on security and privacy. This requires addressing several key challenges:

• The need to streamline security and privacy processes, guardrails and controls for existing and new AI implementations.

• Ensuring compliance with emerging AI laws, regulatory guidance, and industry frameworks.

• Reviewing and addressing data handling practices of third-party AI tools.

• Ensuring employee AI use aligns to the company’s acceptable use policies, avoiding the risk of Shadow AI.

• Oversight by the Executive team and Board of Directors as part of the company’s risk management framework.

Solution

LastPass engaged StackAware to develop a comprehensive AI governance program. Together we leveraged:

• National Institute of Standards and Technology (NIST) AI Risk Management Framework (RMF)

• Open Web Application Security Project (OWASP) Top 10 risks for Large Language Models (LLM)

• System and Organization Controls (SOC) 2

• ISO 27001

• ISO 27701

This collaboration focused on several key areas:

• AI risk assessment. StackAware assessed LastPass’ systems which make use of AI, identifying potential areas for improvement and providing actionable recommendations aligned with best practices and the frameworks described above.

• Vendor review. We jointly completed a thorough review of the vendors used by LastPass - including the underlying AI systems and models - to understand whether and how they would process LastPass’ or its customers’ data with AI.

• Streamlining security and privacy reviews. In addition to due diligence on existing vendors, StackAware improved the security and privacy review process for both commercial and open-source AI systems and models. This reduced friction and accelerated timely deployment while maintaining security.

• Governance development. With StackAware’s help, LastPass built robust AI governance policies and processes, addressing data handling, secure development and ethical AI use.

• Employee training. After the governance framework was in place, LastPass leveraged StackAware to develop training around common risks associated with AI use and how to mitigate them using LastPass processes and procedures.

Results

At the end of the engagement with StackAware, LastPass had:

• A structured approach to identify, assess, and mitigate AI-related risks.

• Compliance framework considerations integrated into the company’s governance program.

• The ability to balance innovation with security, allowing LastPass to move forward with AI initiatives confidently.

"As we embrace AI, ensuring its secure and responsible use is paramount," said Mario Platt, Chief Information Security Officer at LastPass. "StackAware has brought deep expertise in AI governance and risk management, helping us implement a structured, standards-aligned approach to identifying and mitigating AI-related risks. Their guidance empowered us to innovate with confidence, while upholding the trust our customers expect."

Christofer Hoff, Chief Secure Technology Officer, added that "AI isn’t just another technology shift—it’s a trust shift. StackAware helped us translate complex compliance frameworks into practical, actionable steps. Their work enabled us to move fast without breaking trust—balancing innovation with the security and privacy our users expect."

With the StackAware engagement complete, LastPass is well prepared to navigate the complex artificial intelligence landscape while upholding its commitments to its customers and ensure the responsible, ethical, safe and secure use of AI while maintaining a stringent focus on security and privacy.

Are you using AI for mission-critical use cases?

Need effective governance and security?

StackAware can help.

Book a call

​

Joseph Breen is the latest guest contributor to Deploy Securely. In this article, he tackles the evolving landscape of cyber insurance with respect to AI governance.

As businesses rush to deploy AI, the risks grow—and cyber insurance isn’t keeping up.

Subscribe now

Coverage for incidents involving autonomous decision-making, generative AI models, or failures tied to algorithmic outputs is by no mean guaranteed. For companies that rely on AI to power customer service, fraud detection, or content generation, this creates a serious blind spot. While organizations may believe they’re covered by cyber policies, ambiguity or even AI-specific carve-outs can leave them exposed in exactly the kinds of scenarios AI introduces.

During our discussion in March 2025, John Czapko, CEO of CyberSecure, highlighted that most insurers still haven’t incorporated AI into their underwriting processes. He explained that while applications remain lengthy—often exceeding 10 pages—they typically include no reference to AI at all. Supplemental documentation about how a company governs its AI use is rarely requested or accepted. This hesitation stems in part from the insurance industry’s reliance on historical data. Underwriters need robust actuarial models to price risk accurately, but with newly-deployed AI systems, such models are largely unavailable. Real-world claims involving AI failures are still too few and varied to create reliable benchmarks, leaving insurers in a state of uncertainty.

Adding to this challenge, a recent National Association of Insurance Commissioners (NAIC) report highlights how the rapid evolution of AI far outpaces insurers’ ability to develop comprehensive coverage frameworks, resulting in significant gaps in policy language and risk assessment. Furthermore, GlobalData research released in February 2025 points out that a pervasive lack of AI expertise within insurance firms is hindering adoption of AI-informed underwriting practices, deepening the disconnect between AI deployment and insurance readiness. Among more than 120 insurance professionals surveyed in the study, nearly 25% (24.4%) cited lack of AI expertise as the primary hurdle, while customer understanding (21.3%), skepticism about AI readiness (17.3%), and trust issues (13.4%) also featured prominently—emphasizing the increasing divide between AI adoption and insurance readiness.

As a result, I haven’t yet seen major insurers roll out formal premium discounts tied to effective AI governance. But that doesn’t mean the industry isn’t moving in that direction.

Drawing parallels: how security controls shaped cyber insurance

The cyber insurance market rewards organizations that implement preventive controls like multi-factor authentication, endpoint detection and response, and internationally recognized certifications. In a 2024 Insicon blog, the company highlights that achieving ISO 27001 in particular leads to significantly lower premiums, as acknowledged by underwriters such as Aon, Marsh, and Chubb. Similarly, a report by the U.S. Government Accountability Office (GAO) highlights how insurers use such criteria to determine coverage and pricing.

I expect insurers to extend this same logic to artificial intelligence. Companies that can clearly demonstrate how they identify, monitor, and mitigate AI-related risks—especially in areas like model transparency, data governance, and automated decision-making—will be better positioned to negotiate favorable policy terms as insurers evolve their underwriting frameworks to address AI-specific exposures.

Complexities of AI risk: from legal uncertainty to model opacity

AI introduces a host of new risks, including:

• Undesirable or unlawful bias

• Regulatory violations

• Intellectual property disputes

• Data misuse

Generative AI, in particular, creates liability challenges because its outputs are probabilistic rather than deterministic, making them inherently unpredictable. It can produce content that is misleading or offensive—triggering clear reputational and legal risks—or content that may violate copyright law, a question at the center of the ongoing New York Times lawsuit against OpenAI. If those outputs cause harm, is the liability on the developer, the deployer, or the trainer? In some jurisdictions—such as under Colorado’s SB-205 or the EU AI Act—legal frameworks are beginning to clarify those roles and responsibilities. But elsewhere, the lines remain blurry, leaving insurers to grapple with how to assess and price these emerging exposures.

Traditional cyber policies were not built with AI in mind. Many insurers are now reevaluating how to handle exposures introduced by systems that operate with limited human oversight. In a December 2023 article on the Insurance Thought Leadership’s website, Christopher Gallo discusses some of the current blockers—specifically, how AI presents “complex risks” that are “difficult to understand and predict,” making it challenging to define coverage terms with confidence. The absence of historical loss data, combined with evolving legal frameworks, only adds to the ambiguity.

Exploring alternatives: captives vs. governance maturity

While claims arising from AI-related incidents likely fall into a gray area in traditional cyber insurance policies—or are even explicitly excluded—companies are beginning to explore alternative risk financing mechanisms, such as captive insurance, to close coverage gaps. However, this approach can be complex and expensive—especially for small and mid-sized businesses.

A more sustainable path? Reduce the actual (and perceived) risk of AI use through demonstrable governance.

The role of standards in building AI trust

This is where AI governance becomes essential—not just for compliance or ethical reasons, but for business continuity and insurability. Frameworks like ISO/IEC 42001, the first global standard for AI management systems, and the NIST AI Risk Management Framework (AI RMF), offer structured approaches for managing AI risk. These frameworks help organizations classify systems by risk level, assess potential impacts, and implement controls to minimize harm. They also emphasize transparency, documentation, and accountability—elements that are critical for both regulators and insurers.

While I don’t know of any insurers offering discounts specifically tied to adoption of ISO 42001 or the NIST AI RMF, that may change. A 2024 report by The Geneva Association suggests that insurers are starting to consider AI governance maturity—such as the presence of human oversight, transparent documentation, and clear accountability structures—as factors that could eventually inform underwriting models. Deloitte has also highlighted growing interest from insurers in understanding how AI is deployed and governed, particularly in high-risk sectors like healthcare and finance.

Preparing for the shift: practical steps for organizations

To prepare, organizations can strengthen their internal AI controls today. Key actions include:

• Conducting model risk assessments to understand where failures could occur.

• Maintaining audit trails that document how models are developed, trained, and updated.

• Engaging legal, compliance, and security teams early in the AI development lifecycle.

• Establishing oversight mechanisms for monitoring AI in production and intervening when things go wrong.

These steps not only reduce operational and reputational risk now—they also create the foundation for demonstrating governance maturity to insurers in the future.

From cyber to AI: defining the future of insurable innovation

In many ways, this mirrors the evolution of cybersecurity insurance. In the early 2010s, insurers were cautious about covering cyber risk due to a lack of data and standard practices. Over time, they began to recognize and reward companies that adopted controls like encryption, multi-factor authentication (MFA), and incident response plans. The same trajectory is now possible for AI—if the industry has the right signals to evaluate.

Ultimately, as AI is a core business function rather than a fringe experiment, the question isn’t whether insurers will react to AI-related risks. It’s how they will do it—and which organizations will be able to qualify for meaningful, affordable coverage. Those that invest early in AI governance may not only protect themselves from current gaps in coverage—they may help define the very standards that shape the future of insurable AI.

Subscribe now

Deploy Securely is sharing this guest post from Daniel Kalinowski, an ethical hacker and the founder of TLBC, a security company. In this piece, he talks about the Model Context Protocol (MCP) and its implications for cybersecurity.

Artificial Intelligence (AI) agents are popping up everywhere—but how do they actually interact with the world? Enter the Model Context Protocol (MCP) introduced by Anthropic in late 2024: an emerging standard for connecting large language models to tools, systems, and data in a structured way. In 2025, we've seen a surge of MCP server releases from giants like Cloudflare, AWS, Microsoft, and others, turning this protocol into a foundation for next-gen automation.

Subscribe now

This blog post is your guide to understanding MCP, why it matters, and the different shades of its ecosystem—from promising innovations to potential pitfalls.

What is MCP?

The Model Context Protocol (MCP) defines how language models and AI agents communicate with external systems. Instead of relying solely on static prompts or Application Programming Interfaces (APIs) with brittle wrappers (hacky temporary solutions made with duct tape), MCP provides a consistent, open format for actionable context—turning models into dynamic operators.

Think of it as the protocol that allows your AI assistant not just to talk about tools, but to use them.

Recent MCP server launches span cloud management, security, enterprise data, and DevOps. They’re enabling everything from serverless infrastructure orchestration with AWS Lambda to querying network logs via Versa’s SASE MCP server.

The good: agentic automation

The real magic of MCP is its ability to integrate with other systems. Some highlights:

• Natural language for DevOps: AWS’s MCP servers let AI agents spin up Lambda functions, monitor ECS (Elastic Container Service) containers, and audit configurations using human-readable commands.

• AI-Native debugging: Cloudflare’s suite of 15 servers makes deep observability information—logs, browser rendering, DNS (Domain Name System) analytics—accessible to agents.

• Enterprise intelligence: Dremio’s Lakehouse MCP server connects agents to governed, real-time enterprise data for analytics or insights generation.

• Security efficiency: Versa’s server lets AI copilots query firewall logs or automate incident response with context-aware prompts.

MCPs aren’t toy demos—they’re in production with companies like Stripe, Atlassian, Sentry, and Webflow.

The bad: complexity and fragmentation

Despite all the enthusiasm, not everything about MCP is perfect:

• Developer overhead: Standing up your own MCP server (or even contributing to open source ones) requires understanding YAML(Yet Another Markup Language)-based schemas, tracing, tool exposure, and authorization flows.

• Versioning pain: As the protocol evolves, keeping your server aligned with the latest model capabilities and conventions can be a moving target. But that’s the nature of every meaningful standard—from HTTP to GraphQL to OAuth—stability comes over time, but early adopters must adapt. If you want to play at the frontier, this kind of version churn is part of the deal.

• Documentation gaps: Many new MCP projects are shipping fast, with minimal public docs—leaving developers to reverse engineer functionality from GitHub examples.

The promise is there. But the learning curve remains steep.

The ugly: trust and safety hazards

This is where things get spicy. Here are a few examples of how things can go wrong:

• Over-permissioned actions: If an MCP server can use powerful tools (like modifying DNS records or restarting containers) without robust checks, a hallucinating agent could wreak havoc. Picture this: an agent messes with a domain's settings, and boom: thousands of people can't get their emails.

• Opaque auditing: Streamable tool calls and auto-traced responses (when input parameters, timestamps, etc. are automatically recorded) are powerful—but unless well-logged and monitored, they could become a blind spot in security.

  • Inaccurate logging of decisions and actions muddies the waters of accountability and makes discipline unenforceable.

  • Logs themselves can become a security risk if stored in an insecure location, e.g. an internet-facing AWS S3 bucket without authentication.

• Supply chain risks: As open-source MCP servers gain traction, they could become vectors for dependency-based attacks—especially when plugged into high-privilege environments.

  • Say you find a handy open-source MCP server on GitHub that connects to your cloud billing dashboard. Looks clean, has a few stars, and saves you a weekend of work—so you plug it into your AI agent and give it production credentials. But what if that server pulls in a third-party dependency compromised by a malicious maintainer last week? Maybe a nested package starts exfiltrating cloud keys or tweaking cost reports. Suddenly, your helpful AI assistant is unknowingly funneling sensitive data out of your environment—because one unverified repo snuck into the stack.

  • Another potential attack vector: a malicious MCP server could prompt inject yours through a line jumping/tool poisoning attack, contaminating the context of your server. Without ever being invoked, this hostile MCP server could pass corrupted tool descriptions to yours and cause unexpected behavior.

In other words: an AI assistant with access to your cloud infrastructure can be both your greatest enabler and your biggest risk.

Reality check

Look, using MCP Servers securely isn't plug-and-play. You've got to roll up your sleeves, thoroughly test the implementation methods, and really understand what's going on under the hood.

A recent post from Invariant Labs explores the scenario where GitHub MCP server code itself might not contain any flaws, but still pose a risk. The context in which the server is used—deployed alongside content consumed by the the Agent—might result in a leak of the content of a private repository to attacker.

Basically, context is super important, right?

Like with the underlying Large Language Models (LLM), a major security hole with MCP is prompt injection. Because this is basically an impossible-to-fix problem, developers of both need to apply guardrails. Here are some:

1. Layered input and output validation

• Why. Filtering user inputs and LLM outputs can block unsafe content and behaviors.

• How. Open-source classifier packages, purpose-built regex (regular expressions), and trusted third-party content moderation APIs.

2. Continuous monitoring and escalation

• Why. Tracking usage, automating security alerts, and enabling human review for abnormal or high-risk interactions.

• How. Detailed logging, clear behavioral thresholds, pre-established alarms, and on-duty HITL (humans-in-the-loop).

3. Modular, configurable, and evolving architecture

• Why. Building guardrails as configurable, updatable, and standalone components keeps them flexible and durable.

• How. Feature flags and microservice deployment models. Not copying and pasting code.

MCP is here to stay—use, with care

The Model Context Protocol has moved beyond hype—in 2025 it’s clear MCP is a cornerstone of modern AI deployments. Whether you're debugging systems with natural language or querying enterprise data in real time, MCP servers offer incredible leverage.

But as with all powerful tools, they require care. Audit what your agents (can) do. Invest in observability. And don’t let excitement outweigh caution.

The good is revolutionary.

The bad is manageable.

The ugly? Avoidable—with the right guardrails.

Subscribe now