This repository holds an example for Sireum Mr. Roboto (Roboto for short).

Roboto takes a demo automation script (e.g., example.md) and runs it using java.awt.Robot to automate GUI interactions such as typing, keyboard shortcuts, mouse clicks, image-based template matching, on-screen overlay notifications, and text-to-speech.

The specification language is in the form of either:

1. Slash (Slang universal shell) script that builds objects defined by the Script Slang types (outputs JSON consumed by the runner); or,

2. Markdown with YAML frontmatter specifying script attributes and OS-specific variables, with each heading (#) defining an action and bullet points (*) specifying commands (e.g., example.md).

sireum roboto run <option>* <path> <arg>*

where <path> is either a .md (Markdown) or .cmd (Slang script) file.

By default, Roboto uses MaryTTS for speech synthesis (local, no API key needed).

sireum roboto run example.md

Define the AZURE_KEY environment variable using one of your Azure account text-to-speech service keys.

sireum roboto run --service azure example.md

Install AWS CLI and configure it (aws configure).

sireum roboto run --service aws example.md

Record the script execution as an MP4 video with synchronized audio (system audio + TTS speech). Requires FFmpeg.

sireum roboto run --record output.mp4 example.md

The recording captures video with synchronized TTS and system audio.

sireum roboto capture <output.png>

This captures the full screen after a 5-second countdown. The captured image can be cropped (e.g., in Preview on macOS) to create template images for clickImage and waitForImage commands.

---
name: "Script Name"
defaultCharDelayMs: 50
defaultActionDelayMs: 2000
vars:
  - key: value
varsMac:
  - key: macValue
varsWin:
  - key: winValue
varsLinux:
  - key: linuxValue
varsLinuxArm:
  - key: linuxArmValue
subst:
  - hamr: Hammer
  - sysml: Sis M L
---

Variables defined in the frontmatter are substituted in command text using $name$ syntax (dollar-sign delimited).

OS-specific variable sections (varsMac, varsWin, etc.) override base vars for the current platform.

Environment variables in variable values are expanded using $ENV_VAR syntax (single $ prefix). Use $$ to escape a literal $ (e.g., $$SIREUM_HOME produces $SIREUM_HOME without expansion).

The subst section defines pronunciation overrides for text-to-speech. When $term$ syntax is used in speak command text, the term is replaced with its subst value before being sent to the TTS engine. This is useful for terms that TTS engines mispronounce (e.g., CamelCase identifiers, abbreviations, domain-specific terms).

subst:
  - hamr: Hammer
  - sysml: Sis M L
  - isolette: Eye-so-let

* speak: Let me demonstrate $hamr$ code generation for the $isolette$ model.

The TTS engine receives: "Let me demonstrate Hammer code generation for the Eye-so-let model."

Each Markdown heading (#) defines an action. The heading text is the action name. Bullet points (*) under a heading are the commands for that action.

HTML comments (<!-- -->) are ignored and can be used to comment out sections.

Commands use the syntax: command[(options)]: arguments

Enter, Escape (or Esc), Tab, Space, Backspace, Delete, Up, Down, Left, Right, Home, End, PageUp, PageDown, F1–F12

Ctrl, Shift, Alt, Meta (or Cmd)

Image paths are resolved relative to the .md file's directory. The similarity parameter (0.0–1.0, default 0.9) controls how closely the screen region must match the template image. xOff and yOff (default 0) offset the click position from the center of the matched region.

Template images with transparent backgrounds (PNG alpha channel) are supported. Transparent pixels (alpha < 128) are ignored during matching, so only the foreground content needs to match. This allows the same template to work across different themes or background colors.

Workflow:

1. Run sireum roboto capture screenshot.png
2. Crop the target UI element from the screenshot (e.g., in Preview on macOS)
3. Optionally, make the background transparent for theme-independent matching
4. Save the cropped image next to the .md file
5. Reference it: clickImage(0.9): my-button.png

Useful for screen recordings of TUI demos where the mouse pointer would otherwise sit on top of the captured frames.

• hideCursor stops Roboto from compositing its cursor overlay onto the recorded frames (drawCursor=false).
• On macOS it additionally spawns a tiny Swift helper that calls CGDisplayHideCursor(CGMainDisplayID()) and blocks on its stdin so the OS cursor itself is hidden system-wide for the duration. The helper balances the hide with CGDisplayShowCursor when its stdin closes (either via showCursor or the JVM shutdown hook on script exit), so the cursor is never left hidden on a crash.
• On non-macOS platforms the helper is skipped and Roboto falls back to parking the OS pointer at the right edge / vertical-middle of the primary screen, which avoids menu-bar / dock / hot-corner triggers.
• showCursor reverses both effects.

Typical use is a single hideCursor near the top of the script — matching showCursor is only needed if a later section actually requires cursor visibility.