简体中文

A real-screen computer-use agent powered by multimodal LLMs. Built on Agent Zero with the Computer (Pointer) profile: see the display like a human, interact with UIs, fill forms, and complete tasks across applications.

Model support: Only DashScope qwen3.5-plus is verified so far. You must set api_base to the correct regional endpoint as well as the API key—see §4.4.

Pointer is an AI assistant for real desktop and browser UIs. It does not rely on spinning up a separate page in a headless browser; it captures what is visible on your monitor, understands on-screen elements, and acts through mouse, keyboard, shortcuts, and composite actions—the same inputs a person uses.

Capabilities include:

• Interactions in web and desktop apps: click, type, scroll, multi-select, drag, and more
• Multi-step flows: login wizards, forms, file pickers, paginated reading and extraction
• Vision: numbered UI overlays plus optional zoomed regions to reduce mis-clicks
• Structured memory: per-task screen extraction, merge, task_done:checkpoint / read, and persisted execution state (see agents/computer/ docs)

Pointer runs as an Agent Zero profile (computer), so tool orchestration, dialogue, and extensions work the same way. The product name is Pointer.

Goal: We aim to fully replace repetitive, physical computer work—the routine clicking, typing, and screen navigation that people do every day—with capable automation. That includes personal daily tasks (your own desktop and browser workflows) and enterprise employees’ day-to-day work (repeated operational steps across standard business apps and web consoles), wherever vision, judgment, and multi-step control are needed.

Example use cases:

• Complex web workflows (login, search, forms, admin consoles, multi-page collection)
• Desktop apps and hybrid work (real windows in the foreground, coexisting with your workflow)
• Repetitive human–computer tasks that need vision + reasoning + long horizons

Video demo:
(TBD: link or embed)

Try it online / pilot access:
Contact [email protected] for trials, demos, or partnerships.

For architecture, file layout, env vars, and tools, see agents/computer/README.md.

• Python: 3.10+ recommended (match project dependencies)
• OS: macOS, Windows, and Linux; Pointer injects OS-specific shortcut references
• Permissions: Screen capture and accessibility / assistive APIs may be required for screenshots and input—grant them per your OS settings

From the repository root:

pip install -r requirements.txt
pip install -r requirements2.txt

requirements2.txt includes LiteLLM and the OpenAI SDK (model calling). Install both files.

Optional for development / tests:

pip install -r requirements.dev.txt

1. Copy or create an env file (e.g. .env) if your workflow uses one.

2. In the Web UI settings or .env, set keys for your provider (see conf/model_providers.yaml), e.g.:

   • OpenAI, OpenRouter, Anthropic, etc.
   • DashScope (Qwen): DASHSCOPE_API_KEY or API_KEY_DASHSCOPE (provider id: dashscope)

3. Set api_base for DashScope (required). LiteLLM does not auto-pick China vs international. In Settings, fill API base for each role that uses DashScope (Chat, Utility, Browser, etc.—field names such as chat model API base, utility model API base, browser model API base), or ensure the value is merged from conf/model_providers.yaml defaults. Use the URL that matches where your key was issued:

   Region	api_base
   China (Beijing)	https://dashscope.aliyuncs.com/compatible-mode/v1
   International	https://dashscope-intl.aliyuncs.com/compatible-mode/v1

   Wrong region + key combination typically causes auth or routing errors.

4. Pick models for Chat, Utility, Browser, etc. Computer use typically needs a vision-capable Utility model for screenshot understanding and extraction.

Tested models (today): Only Qwen 3.5 Plus on DashScope has been validated end-to-end with Pointer. Use provider dashscope, the correct regional api_base (see table above), DASHSCOPE_API_KEY (or API_KEY_DASHSCOPE), and model name qwen3.5-plus (or the exact id from your DashScope console). Chat, Utility, and other roles may all need this stack until broader compatibility is confirmed. Other providers and model IDs are untested and may not work.

Set Settings → workdir. Pointer writes under it:

• computer/snapshots/ — screenshots and debug images
• computer/extract_data/, computer/task_done/, computer/execution_checkpoint/ — extraction and task state

Avoid pointing workdir at read-only or ephemeral locations unless intentional.

Pointer calls an HTTP annotation service to produce numbered UI boxes (default COMPUTER_ANNOTATE_API_BASE, path /api/v1/annotate/all).

(TBD: how to deploy the service, Docker image, or doc links)

Common environment variables (details in agents/computer/README.md):

python run_ui.py

(TBD: default port, HTTPS, reverse-proxy examples)

1. Lightweight RPA — Web data extraction, chat-driven tasks, and visiting social or content sites (e.g. Xiaohongshu, X, Facebook) where work stays mostly in the browser or a small set of familiar UIs.

2. Cross-application work — Uploading and downloading files across apps, delivering results in chat when a task completes, and auto-filling forms that span multiple programs or sites.

3. Research — Work on professional or domain data that is public but requires login to access; produce and hand off research outputs—for example, legal case research and similar deep dives.

4. Digital coworkers — Full loop: receive a task → plan → execute on the machine → deliver results, aimed at recurring operational work patterned after roles such as developers, HR, and finance.

Pointer has full control of the computer (mouse, keyboard, system-level shortcuts). With bad prompts, model errors, or ambiguous goals, it may:

• Delete or corrupt files and important data
• Trigger unintended payments, emails, or account changes
• Leak privacy (screen content is sent to your model provider)

Use only in controlled environments, with backups and least privilege; high-stakes production use requires human oversight and auditing.
You are solely responsible for any direct or indirect damage from using this software.

• Email: [email protected]
• Feedback, trial requests, partnerships, custom work: [email protected]

See LICENSE in the repository root (if it differs from upstream Agent Zero, the file in this repo prevails).

Pointer builds on Agent Zero; Computer / Pointer code lives under agents/computer/. Thanks to Agent Zero and the broader open-source ecosystem.