There's a second interface running underneath every iOS app. Built for VoiceOver and the millions who depend on it, the accessibility layer is the plumbing beneath the UI. Every control, every action, every state, described in a semantic map we keep up to date under the pixel polish.

In practice, coverage varies. VoiceOver users notice the gaps.

Button Heist lets AI agents in through those pipes, and gives them full control. Link one framework into your debug build and the agent works the interface from the inside. No coordinate math, no screenshot parsing. The exact same APIs VoiceOver uses. It activates a login button by name, calls increment on a stepper, triggers a "Delete" custom action directly.

Every interaction doubles as an accessibility audit: if the agent can't find a control, neither can VoiceOver.

The heist works because the infrastructure was already in place. A language interface built for people to navigate apps by meaning. Turns out agents thrive there too.

Link TheInsideJob to your debug target. It starts a local TCP server via ObjC +load. No setup code. DEBUG only, stripped from release builds.

import SwiftUI
import TheInsideJob

@main
struct MyApp: App {
    var body: some Scene {
        WindowGroup { ContentView() }
    }
}

Same embed pattern as Reveal or FLEX. Add the Info.plist entries so Bonjour can advertise:

<key>NSLocalNetworkUsageDescription</key>
<string>This app uses local network to communicate with the element inspector.</string>
<key>NSBonjourServices</key>
<array>
    <string>_buttonheist._tcp</string>
</array>

Install the CLI and MCP server:

brew install RoyalPineapple/tap/buttonheist

Then add the MCP server to your project's .mcp.json:

{
  "mcpServers": {
    "buttonheist": {
      "command": "buttonheist-mcp",
      "args": []
    }
  }
}

This exposes 23 tools to your agent: get_interface, activate, type_text, run_batch, get_screen, and more. The agent discovers your app via Bonjour automatically:

Agent: "I need to log the user in"

→ get_interface
  textfield_email, textfield_password, button_login (12 elements)

→ run_batch([type_text into textfield_email, activate button_login])
  step 1: value → "[email protected]" ✓
  step 2: screen changed: login gone, dashboard appeared ✓

The agent stays focused on the task, not on driving the app.

cd ButtonHeistCLI && swift build -c release && cd ..
BH=./ButtonHeistCLI/.build/release/buttonheist

$BH list                                                  # Discover devices (WiFi + USB)
$BH session                                               # Interactive REPL
$BH activate --identifier loginButton                     # Activate an element
$BH action --name "Delete" --identifier cell_row_3        # Named custom action
$BH type --text "Hello" --identifier nameField            # Type into a field
$BH scroll --direction down --identifier scrollView       # Scroll one page
$BH scroll_to_visible --identifier targetElement          # Scroll until visible
$BH screenshot --output screen.png                        # Capture screenshot
$BH record --output demo.mp4 --fps 8 --scale 0.5         # Record with touch overlay

The session REPL accepts both JSON and shorthand: tap loginButton, type "hello", scroll down list, screen.

But wait, there's more: gestures, recording, pasteboard, scroll modes, multi-device. See the API Reference.

The coordinate-based approach reads the accessibility tree, extracts element frames, and throws the rest away. The agent works with geometry, not meaning. Every action requires re-reading the full tree to know what happened.

Button Heist works from the inside, the same position VoiceOver occupies. The framework lives in your app. It doesn't snapshot the hierarchy and discard it. It holds the live tree and sees every change as it happens.

Three things follow from being inside:

After every command, Button Heist diffs the accessibility hierarchy and returns what moved: an interface delta. Tap "Login" and the response carries exactly which elements disappeared and which appeared:

{
  "success": true,
  "method": "activate",
  "interfaceDelta": {
    "kind": "elementsChanged",
    "elementCount": 14,
    "removed": ["button_login", "textfield_password", "textfield_email"],
    "added": [
      {"heistId": "header_dashboard", "label": "Dashboard", "traits": ["header"]},
      {"heistId": "button_settings", "label": "Settings", "traits": ["button"]}
    ]
  }
}

Login screen gone, dashboard appeared, new elements ready to target. Value updates carry the property change inline: old value, new value, which element. When nothing changes, the delta says "noChange" and the agent pivots immediately.

The agent doesn't need to re-read the screen. The next decision starts from where the last one landed.

The crew inside keeps watch between jobs too. Every response carries a background delta: what changed in the UI while the agent was thinking. Content loaded, a dialog appeared, an animation settled. No stale intel.

Each command can carry an expect, a declaration of what should happen. The framework checks the delta against the expectation and reports pass/fail inline:

{
  "command": "activate",
  "target": {"heistId": "button_login"},
  "expect": "screen_changed"
}

Response: {"expectation": {"met": true, "expectation": "screenChanged"}}.

Three tiers: screen_changed (new view controller), elements_changed (anything in the hierarchy shifted), or element_updated with specific property checks. When an expectation fails, the response carries what actually happened.

The agent says what it expects. The framework says whether that happened.

An agent that trusts its feedback loop can commit to a whole sequence at once. run_batch sends ordered steps in a single round trip. Each one gets its own delta and expectation check. If a step fails, the batch stops. The agent never pushes forward with bad state:

{
  "command": "run_batch",
  "steps": [
    {"command": "type_text", "target": {"heistId": "textfield_email"}, "text": "[email protected]",
     "expect": {"element_updated": {"heistId": "textfield_email", "property": "value", "newValue": "[email protected]"}}},
    {"command": "activate", "target": {"heistId": "button_submit"}, "expect": "screen_changed"}
  ]
}

Two actions, two assertions, one round trip. If the email field doesn't update, the batch stops there.

Deltas, expectations, and batching, each one enabling the next. That's the compound advantage.

Because every action carries a semantic target and a structured result, Button Heist can record an agent's session as a replayable .heist file. Each step is captured as a semantic matcher — label, traits, identifier — not coordinates or ephemeral IDs. The matcher targets the accessibility contract, not transient UI state, so it stays stable across runs.

Replay re-executes each step through the same dispatch path. If an element can no longer be found by its accessibility properties, the test fails. That failure means one thing: an accessibility contract broke. The label changed, a trait disappeared, a custom action was removed. JUnit XML output (--junit) puts these into CI.

The agent did its job. The test suite wrote itself. And because matchers are semantic, the same recording works on any device, any screen size, any orientation. A coordinate-based recording breaks the moment the layout shifts. A semantic recording breaks only when the accessibility interface breaks.

But the deepest advantage isn't speed. The agent and a VoiceOver user navigate the same hierarchy, interacting with the exact same elements. A coordinate-based tool can tap a button with broken accessibility and never notice. Button Heist can't. If VoiceOver can't see a control, neither can the agent. Every session is an accessibility audit, whether you asked for one or not.

Accessibility bugs stick around because the people who report them rarely have the leverage to get them prioritized. When an agent hits the same bug, it blocks automation and gets fixed. VoiceOver users benefit.

Agents already write our code. When they inspect what they've built, they see it through the accessibility layer. Make it good for them, and you've made it good for everyone.

Tested against a coordinate-based MCP server using the same model, same app, same tasks. 96 trials across 16 UI automation tasks. Both tools ran against the same app using standard iOS design patterns: forms, navigation, lists, controls.

2.4x faster, 3.1x fewer turns, 3.1x cheaper. The gap scales with complexity:

The difference is in what the agent gets back. A coordinate-based tap:

→ tap(x: 201, y: 456)
← "Tapped successfully"

The same action through Button Heist:

→ activate(heistId: "large_button")
← elements changed
  + text_size_large_staticText "Text Size, Large"
  - text_size_medium_staticText
  ~ large_button: traits "button" → "button, selected"
  ~ medium_button: traits "button, selected" → "button"

"Tapped successfully." That's the whole response. The agent has to re-read the entire screen to find out what happened. The delta reveals it all: which properties changed, which elements appeared and disappeared, the entirety of the new state. No follow-up needed.

That difference compounds. Every action without a delta costs a full tree read. Over a 50-action workflow, that's 50 extra round trips filling the context window. On our longest benchmark, Button Heist finished in under 8 minutes. The coordinate-based tool needed 20.

Full methodology and per-task data: docs/BENCHMARKS.md

That's the job. What follows is the crew, the blueprints, and the fine print.

Every heist needs a team.

• Xcode with Swift 6 package support
• iOS 17+ / macOS 14+
• git submodule update --init --recursive
• Tuist

git submodule update --init --recursive
tuist generate
open ButtonHeist.xcworkspace

ButtonHeist/
├── ButtonHeist/Sources/          # Core frameworks (TheScore, TheInsideJob, ButtonHeist)
├── ButtonHeistMCP/               # MCP server (Swift Package)
├── ButtonHeistCLI/               # CLI tool (Swift Package)
├── TestApp/                      # SwiftUI + UIKit test applications
├── AccessibilitySnapshotBH/      # Git submodule (hierarchy parsing)
├── docs/                         # Architecture, API, protocol, auth, USB docs
│   └── dossiers/                 # Per-module technical documentation

1. Both devices on the same network
2. TheInsideJob framework linked to your target
3. Info.plist has the _buttonheist._tcp Bonjour service entry
4. iOS local network permission accepted

1. Device connected: xcrun devicectl list devices
2. App running on device
3. IPv6 tunnel visible: lsof -i -P -n | grep CoreDev

• App has visible UI on screen
• Root view is accessible to UIAccessibility
• Run buttonheist get_interface and check the element count

Integrating into your app? Start with the API Reference and Quick Start above.

Connecting an agent? See the Wire Protocol and MCP Server.

Understanding the architecture? Read Architecture and Crew Dossiers.

All docs: API ・ Architecture ・ Wire Protocol ・ Auth ・ USB ・ Bonjour Troubleshooting ・ Reviewer's Guide ・ Crew Dossiers

Apache License 2.0. See LICENSE.

• KIF (Keep It Functional). TheSafecracker's touch synthesis is built on KIF's pioneering work in programmatic iOS UI interaction.
• AccessibilitySnapshot. Used for parsing UIKit accessibility hierarchies (via our fork AccessibilitySnapshotBH).