ClawGUI-Agent: Personal Phone GUI Assistant

English | 中文

ClawGUI-Agent is the deployment module of ClawGUI. Built on OpenClaw and powered by nanobot, it provides two core capabilities: GUI phone control and one-command evaluation. For phone control, it drives a Vision-Language Model through a closed-loop "screenshot → reasoning → action" cycle to autonomously complete tasks on Android, HarmonyOS, and iOS devices — accessible from Feishu, QQ, Telegram, and 12+ other chat platforms. For evaluation, a single natural-language command triggers the full ClawGUI-Eval pipeline: environment check, multi-GPU inference, judging, and metric reporting.

• Key Features
• Architecture
• How the Agent Works
• Quick Start
  • Requirements
  • 1. Installation
  • 2. Initialize and Edit Configuration
  • 3. Connect Android Device
  • 4. Configure Chat Platforms (optional)
• Run
• ClawGUI-Eval Evaluation
• GUI Phone Control
  • Web UI
  • Memory System
  • Supported GUI Models
• Directory Structure
• License

• nanobot Integration — Remotely control phones from 12+ chat platforms including Feishu / DingTalk / Telegram / Discord / Slack / QQ, issue tasks anytime anywhere
• GUI Phone Control — Powered by OpenClaw, AI autonomously captures screenshots, understands the screen, and performs tap/swipe/type GUI actions to complete complex tasks
• ClawGUI-Eval Integration — Built-in ClawGUI-Eval evaluation skill, launch GUI Grounding model benchmarks with natural language (environment check → multi-GPU inference → judging → metric calculation), with automatic progress monitoring and result comparison against official baselines
• Multi-Model Support — Compatible with AutoGLM, Qwen VL, UI-TARS, MAI-UI, GUI-Owl and more VLMs, connected via OpenAI-compatible API
• Personalized Memory — Automatically learns user preferences (contacts, frequently used apps, habits), with a vector-search-based persistent memory system
• Real-time Episode Recording — Each task execution (screenshots + model outputs + actions) is saved as a structured episode, enabling replay and dataset construction
• Web UI — Gradio-based web interface for device management, task execution visualization, manual takeover, memory management and more

Understanding the execution loop helps with configuration and debugging. PhoneAgent.run() in phone_agent/agent.py follows this cycle for each task:

1. Screenshot — Capture the current screen via ADB (screencap), HDC, or XCTest depending on the device backend.
2. Memory retrieval — Query the vector memory store for relevant memories from past interactions (contacts, app knowledge, user preferences). Top-k similar memories are appended to the system context.
3. History construction — Assemble the multi-turn conversation history: each past step contributes a (user: screenshot + instruction, assistant: reasoning + action) pair, up to history_length steps back.
4. VLM call — Send the prompt (system prompt + history + current screenshot + task instruction) to the configured GUI model via OpenAI-compatible API.
5. Action parsing — Extract the structured action from the model output. Different models use different output formats (autoglm, uitars, qwenvl, maiui, guiowl adapters in phone_agent/model/adapters.py).
6. Coordinate normalization — Convert the model's output coordinates to absolute device pixels. AutoGLM uses [0, 1000] normalized coordinates; UI-TARS uses absolute pixel coordinates in smart_resize space; Qwen-VL uses absolute pixels; MAI-UI uses [0, 1000].
7. Action execution — Send the action to the device backend: tap, long-press, swipe, type, home, back, or task-complete. Each action type has a dedicated handler in phone_agent/actions/.
8. Trace recording — If traceEnabled=True, append the screenshot, reasoning, and action to the episode tracer for later replay or training data export.
9. Memory update — After task completion, extract contact names, app knowledge, and user habits from the conversation and upsert them into the vector store with deduplication.

This loop runs until the model outputs a terminate or answer action, or max_steps is reached.

• Python: ≥ 3.11
• Package Manager: uv (recommended) or conda + pip

Assuming you have cloned the ClawGUI project and are in the root directory:

cd clawgui-agent

# Create virtual environment
uv venv .venv --python 3.12

# Activate
source .venv/bin/activate

# Install phone_agent
uv pip install -e .

# Install nanobot
uv pip install -e nanobot/

cd clawgui-agent

# Create conda environment
conda create -n opengui python=3.12 -y
conda activate opengui

# Install phone_agent
pip install -e .

# Install nanobot
pip install -e nanobot/

Run the onboard wizard to generate default config:

nanobot onboard

Then edit ~/.nanobot/config.json. Here is a reference configuration:

We recommend using autoglm-phone as the external GUI model for phone control.

{
  "agents": {
    "defaults": {
      "workspace": "/path/to/ClawGUI",
      "model": "glm-5",
      "provider": "zhipu",
      "maxTokens": 8192,
      "contextWindowTokens": 131072,
      "temperature": 0.1,
      "maxToolIterations": 40
    }
  },
  "providers": {
    "zhipu": {
      "apiKey": "YOUR_ZHIPU_API_KEY",
      "apiBase": "https://open.bigmodel.cn/api/paas/v4/"
    },
    "openrouter": {
      "apiKey": "YOUR_OPENROUTER_API_KEY",
      "apiBase": "https://openrouter.ai/api/v1"
    }
  },
  "tools": {
    "gui": {
      "enable": true,
      "deviceType": "adb",
      "deviceId": null,
      "maxSteps": 50,
      "useExternalModel": true,
      "guiBaseUrl": "https://openrouter.ai/api/v1",
      "guiApiKey": "YOUR_OPENROUTER_API_KEY",
      "guiModelName": "autoglm-phone",
      "promptTemplateLang": "en",
      "promptTemplateStyle": "autoglm",
      "traceEnabled": false,
      "traceDir": "gui_trace"
    },
    "exec": {
      "enable": true,
      "timeout": 60
    }
  }
}

Important: workspace Path Setting

Set workspace to the ClawGUI project root — the directory that contains both clawgui-agent/ and clawgui-eval/. The built-in evaluation skill uses this path to locate the evaluation framework. For example, if your project lives at /home/user/ClawGUI, set workspace to "/home/user/ClawGUI".

The controlled phone must be connected (e.g. via USB) to the server machine where ClawGUI-Agent is installed.

Option A: Install via package manager

macOS (recommended: brew):

brew install android-platform-tools

Linux:

sudo apt install android-tools-adb   # Ubuntu/Debian

Windows: See this blog tutorial to download and configure PATH.

Option B: Manual download

Download the official ADB platform-tools and extract it, then add it to your PATH:

macOS / Linux:

# Assuming extracted to ~/Downloads/platform-tools
export PATH=${PATH}:~/Downloads/platform-tools

Windows: Add the extracted directory (e.g. C:\platform-tools) to the system PATH environment variable.

1. Enable Developer Mode: Go to Settings > About Phone > Build Number, tap rapidly ~10 times until you see "You are now a developer"
2. Enable USB Debugging: Go to Settings > Developer Options > USB Debugging, toggle it on (some devices may require a restart)
3. Verify connection:

adb devices

# Expected output:
# List of devices attached
# <your_device_id>   device

ADB Keyboard is used for text input. Download ADBKeyboard.apk and install:

adb install ADBKeyboard.apk
adb shell ime enable com.android.adbkeyboard/.AdbIME

Note: This step is optional. The framework will auto-detect and prompt installation when needed.

See the Open-AutoGLM device connection guide.

To remotely control the phone via chat platforms, enable the corresponding platform in channels within config.json and fill in credentials.

4. Configure in ~/.nanobot/config.json:

"feishu": {
  "enabled": true,
  "appId": "YOUR_APP_ID",
  "appSecret": "YOUR_APP_SECRET",
  "encryptKey": "",
  "verificationToken": "",
  "allowFrom": ["*"],
  "groupPolicy": "mention"
}

allowFrom set to ["*"] allows all users. To restrict, provide a list of user Open IDs. groupPolicy set to "mention" means the bot only responds when @mentioned in groups.

1. Go to QQ Open Platform and create a bot application
2. Obtain the App ID and Secret
3. Configure in ~/.nanobot/config.json:

"qq": {
  "enabled": true,
  "appId": "YOUR_APP_ID",
  "secret": "YOUR_SECRET",
  "allowFrom": ["*"]
}

nanobot also supports Telegram, Discord, Slack, DingTalk, WeCom, WhatsApp, Email and 12+ more platforms. Set "enabled": true in the corresponding channels field and fill in credentials.

Start the nanobot gateway service:

nanobot gateway

Once started, you can send messages on configured chat platforms (e.g. Feishu) to control the phone:

Open WeChat and send "I'll be late" to Zhang San

nanobot will invoke the gui_execute tool, automatically capturing screenshots → VLM reasoning → executing phone actions in a loop until the task is completed.

ClawGUI-Agent includes a built-in ClawGUI-Eval skill that turns natural language into a complete benchmark run — from GPU environment check through multi-GPU inference, judging, and metric reporting — without writing a single script.

1. workspace correctly set: workspace in config.json points to the ClawGUI root directory (see configuration above)
2. ClawGUI-Eval environment installed: Follow ClawGUI-Eval README to install and download data
3. GPU available: Inference requires NVIDIA GPUs
4. (Recommended) Install FlashAttention-2: pip install flash-attn --no-build-isolation — the framework falls back to SDPA automatically if not installed, but precision may be slightly lower

Simply say it in a nanobot conversation:

Benchmark qwen3vl 2b model on screenspot-pro

Run uivision and osworld-g evaluation with MAI-UI-8B

nanobot will automatically:

1. Environment Check — Check GPU, CUDA, FlashAttention-2, data integrity
2. Inference — Generate run scripts from templates, launch multi-GPU parallel inference in background, monitor progress in real-time
3. Judging — Automatically select and run the corresponding judge script
4. Metric Calculation — Automatically select and run the corresponding metric script
5. Result Report — Present accuracy, sub-category breakdowns, and comparison against official baselines

Supported Benchmarks: ScreenSpot-Pro, ScreenSpot-V2, UIVision, MMBench-GUI, OSWorld-G, AndroidControl

The following features are part of ClawGUI-Agent's phone/device control capabilities, driven by the gui_execute tool.

You can also invoke the GUI agent directly via command line:

python main.py \
  --base-url https://open.bigmodel.cn/api/paas/v4/ \
  --model autoglm-phone \
  --apikey <YOUR_API_KEY> \
  --max-steps 100 \
  --lang cn \
  "Open QQ Music, play Justin Bieber's Baby and add it to favorites. If it is already favorited, just play it. After it starts playing, pause it, then go back and play Bieber's Love Me."

In addition to chat platform control, you can use the Web UI directly:

python webui.py

Opens at http://localhost:7860 by default, featuring:

• Device Management: Connect/disconnect devices, view device status
• Task Execution: Enter task descriptions, watch screenshots and AI reasoning in real-time
• Manual Takeover: Switch to manual control for scenarios like CAPTCHAs
• Memory Management: View/edit/clear memory data
• Configuration Panel: Graphical model parameter settings

The framework includes a built-in personalized memory system (phone_agent/memory/). After each completed task, the agent extracts structured facts from the conversation — contact names and relationships, app-specific knowledge, user habits and preferences — and upserts them into a persistent store as JSON records with numpy vector embeddings. On subsequent tasks, the top-k most semantically similar memories are retrieved and injected into the system context, letting the agent recognize "Zhang San" as the user's colleague or know which music app the user prefers. Duplicate memories are detected and merged rather than accumulated, keeping the store lean. Multi-user isolation is supported via per-user namespaces.

The framework supports multiple Vision-Language Models via an adapter pattern:

All models are connected via OpenAI-compatible API and can be deployed locally with vLLM / SGLang, or connected to cloud services such as Zhipu BigModel, Alibaba Cloud Bailian, or OpenRouter.

ClawGUI-Agent/
├── main.py                      # CLI entry point
├── webui.py                     # Gradio Web UI entry point
├── ios.py                       # iOS CLI entry point
├── setup.py                     # Package setup
├── requirements.txt             # Python dependencies
│
├── phone_agent/                 # Core phone automation package
│   ├── agent.py                 # PhoneAgent main class (screenshot→VLM→action loop)
│   ├── agent_ios.py             # IOSPhoneAgent class
│   ├── device_factory.py        # Device type factory (ADB / HDC / XCTest)
│   ├── tracer.py                # Episode execution tracer
│   ├── config/                  # Configuration & prompts (8 template files)
│   ├── model/                   # Model clients & adapters (5 VLM adapters)
│   ├── adb/                     # Android ADB device control
│   ├── hdc/                     # HarmonyOS HDC device control
│   ├── xctest/                  # iOS XCTest device control
│   ├── actions/                 # Action handlers (tap, swipe, type, etc.)
│   └── memory/                  # Personalized memory system (vector store)
│
├── nanobot/                     # nanobot subproject
│   ├── nanobot/                 # nanobot core package
│   │   ├── agent/               # Agent core + GUI tool
│   │   ├── channels/            # 12+ chat platform integrations
│   │   ├── providers/           # 20+ LLM provider adapters
│   │   └── skills/              # Pluggable skills (gui-mobile, clawgui-eval)
│   ├── pyproject.toml
│   └── README.md
│
├── examples/                    # Usage examples
└── scripts/                     # Deployment & verification scripts

This project is licensed under the Apache License 2.0. The nanobot subproject is licensed under the MIT License.