ClawGUI-Eval: Standardized GUI Grounding Evaluation

English | 中文

• Overview
• Architecture
• Installation
• Download Data
• Project Structure
• Supported Benchmarks & Models
• Reproduction Tips
• Quick Start
• Script Parameters
• Adding a New Model
• Data Format
• Reproduction Results
• Roadmap
• License

ClawGUI-Eval is the evaluation module of ClawGUI. GUI grounding evaluation is harder to reproduce than it looks: prompt order, coordinate systems, temperature, and image resolution all interact to shift numbers by several points. ClawGUI-Eval pins all of these choices per model and adopts a three-stage pipeline — Infer → Judge → Metric — to evaluate how accurately a model can locate UI elements based on natural language instructions. The result is a 95.8% reproduction rate against official baselines, making cross-paper comparisons meaningful.

Key Features:

• Dual backend support — Local GPU via transformers or remote API via OpenAI-compatible endpoints
• 6 benchmarks — ScreenSpot-Pro, ScreenSpot-V2, UIVision, MMBench-GUI, OSWorld-G, AndroidControl
• 12+ models — Qwen3-VL, Qwen2.5-VL, UI-TARS, MAI-UI, GUI-G2, UI-Venus, Gemini, Seed 1.8, Kimi K2.5, and more
• Multi-GPU & multi-thread — NUM_GPUS processes launched via Python multiprocessing, each pinned to one GPU via CUDA_VISIBLE_DEVICES. Shard files are automatically split and merged; interrupted runs resume from the last completed shard.
• Easily extensible — Add new models by inheriting a simple base class; shared architectures (e.g. UI-TARS extends Qwen2.5-VL) reuse parent model loading and only override prompt building and output parsing
• Faithful reproduction — Comprehensive reproduction results with detailed official vs. reproduced comparisons (see details)
• Frontier model evaluation — Successfully reproduced Gemini 3.0 Pro and Seed 1.8 official results on ScreenSpot-Pro using a Zoom paradigm (2-stage crop-then-ground: Gemini uses 25% crop tiles, Seed uses 50% crop tiles), and added Gemini 3.1 Pro evaluation
• ClawGUI-Agent integration — Pair with ClawGUI-Agent to launch the full evaluation pipeline with a single natural language command (env check → inference → judging → metrics). See ClawGUI-Agent README for setup details

Docker eliminates dependency conflicts and makes it easy to share exact evaluation environments.

Prerequisites: NVIDIA Container Toolkit

cd ClawGUI/clawgui-eval

# Build the image (first build is slow due to flash-attn compilation)
docker build -t clawgui-eval .

Then create a .env file to point at your data and model directories:

# .env
DATA_DIR=/data/clawgui-eval/data
IMAGE_DIR=/data/clawgui-eval/image
OUTPUT_DIR=/data/clawgui-eval/output
MODEL_DIR=/data/models           # HuggingFace model cache or local weights

Run any inference script inside the container:

# Inference
docker compose run clawgui-eval \
    bash scripts/infer/transformers/qwen3vl_run_transformers.sh

# Judge
docker compose run clawgui-eval \
    bash scripts/judge/screenspot-pro_run_judge.sh

# Metric
docker compose run clawgui-eval \
    bash scripts/metric/run_metric_screenspot_pro.sh

Note: Edit MODEL_PATH inside the shell scripts to point to /models/<your-model-dir> (the container-side path of MODEL_DIR).

cd ClawGUI/clawgui-eval
conda create -n opengui python=3.12 -y
conda activate opengui
pip install -r requirements.txt
# Recommended: FlashAttention-2 for better precision (falls back to SDPA if not installed)
pip install flash-attn==2.8.1 --no-build-isolation
# Optional: vLLM support
pip install vllm==0.11.0

💡 Tip: If building flash-attn from source is too slow, you can download a prebuilt wheel from the flash-attn releases page and install it directly.

Benchmark images and data files are hosted on Hugging Face and ModelScope. Download them before running evaluations.

From Hugging Face:

pip install -U huggingface_hub

# If you have trouble accessing HF, use the mirror:
# export HF_ENDPOINT=https://hf-mirror.com

huggingface-cli download johnzqlu/clawgui-eval --repo-type dataset --local-dir .

From ModelScope:

pip install -U modelscope

modelscope download --dataset Matrix0602/clawgui-eval --local_dir .

Then extract the archives under the clawgui-eval/ directory:

cd clawgui-eval
unzip image.zip
unzip data.zip
unzip output.zip

⚠️ Important: All zip files (image.zip, data.zip, output.zip) must be extracted under the clawgui-eval/ directory to ensure the relative paths resolve correctly.

clawgui-eval/
├── 📄 main.py                          # Inference entry point
├── 📂 inference/                        # Model inferencers
│   ├── base_inferencer.py               # Abstract base class
│   ├── qwen3vl_inferencer.py            # Qwen3-VL
│   ├── qwen25vl_inferencer.py           # Qwen2.5-VL
│   ├── maiui_inferencer.py              # MAI-UI
│   ├── stepgui_inferencer.py            # StepGUI
│   ├── guiowl15_inferencer.py           # GUI-Owl 1.5
│   ├── guig2_inferencer.py              # GUI-G2
│   ├── uitars_inferencer.py             # UI-TARS (extends Qwen2.5-VL)
│   ├── uivenus15_inferencer.py          # UI-Venus 1.5 (extends Qwen3-VL)
│   ├── uivenus_inferencer.py            # UI-Venus (extends GUI-G2)
│   ├── gemini_inferencer.py             # Gemini (API, optional Zoom)
│   ├── seed_inferencer.py               # Seed 1.8 (API, optional Zoom)
│   └── kimi_inferencer.py               # Kimi K2.5 (API, optional Zoom)
├── 📂 judge/                            # Judgment module
│   ├── base_judge.py                    # Abstract base class
│   ├── grounding_judge.py               # Point-in-box judge (most benchmarks)
│   ├── osworld_g_judge.py               # OSWorld-G judge (bbox/polygon/refusal)
│   └── androidcontrol_judge.py          # AndroidControl judge (multi-action)
├── 📂 metric/                           # Metric calculation
│   ├── base_metric.py
│   ├── screenspotpro_metric.py
│   ├── screenspotv2_metric.py
│   ├── mmbenchgui_metric.py
│   ├── osworldg_metric.py
│   ├── uivision_metric.py
│   └── androidcontrol_metric.py
├── 📂 data/                             # Benchmark data & prompt injection
│   ├── convert_any_models.py            # Prompt injection script
│   └── *.json                           # Base & model-specific data files
├── 📂 scripts/
│   ├── infer/
│   │   ├── transformers/                # Local GPU inference scripts
│   │   ├── api/                         # API inference scripts
│   │   └── vllm_depoly/                 # vLLM server deployment
│   ├── judge/                           # Judge scripts (one per benchmark)
│   └── metric/                          # Metric scripts
├── 📂 image/                            # Benchmark images (downloaded)
└── 📂 output/                           # Inference & judge output

We have also reproduced GUI grounding results for frontier models on ScreenSpot-Pro using the Zoom paradigm (crop-then-ground). For details on the Zoom pipeline, see the MAI-UI blog: A Practical Guide to GUI Grounding for Frontier Models.

📐 Coordinate Systems:

• Absolute — Output is in raw pixel coordinates of the original (or smart_resize'd) image
• [0, 1000] — Output is normalized to a 1000×1000 coordinate space, then mapped back to the original image
• [0, 1] — Output is a ratio in [0, 1] relative to the original image dimensions
• [0, 999] — Similar to [0, 1000] but with a 999 divisor

Two backends are supported:

bash scripts/infer/transformers/qwen3vl_run_transformers.sh

# 1. Deploy vLLM service first
bash scripts/infer/vllm_depoly/vllm_serve.sh

# 2. Run inference
bash scripts/infer/api/qwen3vl_run_api.sh

# Kimi K2.5 API
bash scripts/infer/api/kimi_run_api.sh

Output is saved to:

output/<experiment_name>/<benchmark>/predictions.jsonl

# GUI Grounding benchmarks
bash scripts/judge/screenspot-pro_run_judge.sh

# AndroidControl benchmark
bash scripts/judge/androidcontrol_run_judge.sh

Each record gets a correct field (true/false). Output:

output/<experiment_name>/<benchmark>/predictions_judge.jsonl

# GUI Grounding benchmarks
bash scripts/metric/run_metric_screenspot_pro.sh

# AndroidControl benchmark
bash scripts/metric/run_metric_androidcontrol.sh

Reports accuracy broken down by platform, UI type, etc.

In addition to the parameters above:

1. Create inference/<name>_inferencer.py, extending BaseInferencer (or an existing inferencer if architectures match).

2. Implement four methods: _init_model(), _build_prompt(), _generate(), _post_process().

3. Register in inference/__init__.py:

   INFERENCER_REGISTRY = {
       ...
       "your_model": YourModelInferencer,
   }

4. Add prompt injection logic in data/convert_any_models.py, then generate data files.

5. Add parsing logic in judge/grounding_judge.py (and osworld_g_judge.py if needed).

6. Create launch scripts under scripts/infer/transformers/ and scripts/infer/api/.

Each input sample must contain the following fields:

A key goal of ClawGUI-Eval is faithful reproduction of officially reported numbers. Below we compare our reproduced results against official baselines across all supported benchmarks.

📂 All inference results are publicly available on our dataset page: 🤗 HuggingFace: johnzqlu/clawgui-eval | 🤖 ModelScope: Matrix0602/clawgui-eval

Criterion: A result is considered successfully reproduced (✅) if the reproduced number meets or exceeds the official number, or the absolute difference is ≤ 2%. - means no official baseline is available.

Open-Source GUI Grounding Reproduction Rate: 44 / 46 cells with official baselines = 95.7%

Frontier Model ScreenSpot-Pro Reproduction Rate: 2 / 2 = 100.0%

Overall Reproduction Rate: 46 / 48 = 95.8%

AndroidControl evaluates offline navigation with multi-action prediction (click, type, scroll, etc.). We currently support Qwen3-VL and Qwen2.5-VL on this benchmark.

Note: Official AndroidControl baselines for these models are not yet publicly available. We will update the comparison once official numbers are released.

• Support ScreenSpot-Pro, ScreenSpot-V2, UIVision, MMBench-GUI, OSWorld-G benchmarks
• Support AndroidControl benchmark (Qwen3-VL, Qwen2.5-VL)
• Transformers & API dual backend inference
• Multi-GPU parallel inference with automatic resume
• Frontier model reproduction (Claude 4.5 Sonnet, Gemini 3.1/3.0 Pro, Seed 1.8) with Zoom paradigm
• Integrate vLLM offline inference (non-server mode)
• Add more GUI-specific models
• GUI offline navigation evaluation (e.g. GUI-Odyssey)

This project is licensed under the Apache License 2.0.