Human Operator

• 

  Human Operator: Winning project at MIT Hard Mode 2026 for the Learn track.

What it does

Human Operator is a human augmentation tool that allows AI to briefly take control of your body to help you learn and do things you normally cannot do. To do this, it uses a Vision-Language Model for human motor control through Electrical Muscle Stimulation (EMS). Vision-based commands are generated via open-ended speech input through the Claude API to control finger and wrist stimulation for intuitive on-body interaction.

How we built it

Prerequisites

• Python 3.10.x
• Conda, Miniconda or other environment setup
• Arduino IDE 2.3.x (for firmware setup)
• Claude API key (from Anthropic)

Hardware Setup

EMS & Control:

• TENS/EMS Gel Electrodes
• Arduino-compatible microcontroller (Arduino Micro recommended)
• 5V Relays for finger control (pinky, middle, index)
• Custom or commercial TENS/EMS unit with electronic control capability

Sensing:

• Camera module (integrated or USB) for POV video capture

Software Setup

1. Clone the repository:

   git clone <repository-url>
   cd Human-Operator
   

2. Create Python environment:

   conda env create -f environment.yml
   conda activate human-operator
   

3. Configure API key:

   cp .env_empty .env
   # Edit .env and add your Anthropic API key:
   # ANTHROPIC_API_KEY=your_api_key_here
   

4. Upload Arduino firmware:

   • Open Arduino IDE
   • Load firmware/human_operator_ems/human_operator_ems.ino
   • Upload to your Arduino board
   • Verify serial communication works (115200 baud)

Run Main Application

Start the real-time EMS control system:

python app.py

This launches:

• Video capture from your camera
• AI processing via Claude API for vision-based motor commands
• Hardware control for EMS stimulation of pinky, middle, and index fingers
• Voice trigger ("Hey Operator") to activate commands

Commands are triggered by physical interaction detected in your POV video.

Manual Stimulation GUI

Test and calibrate EMS parameters with the GUI:

python utils/stimGUIfingersHardMode.py

This tool allows you to:

• Manually trigger finger stimulation (pinky, middle, index)
• Adjust amplitude, frequency, and pulse width
• Visualize stimulation parameters in real-time
• Perfect for hardware calibration and testing

System Architecture

app.py (Main Application)
    ├── Video Capture (OpenCV)
    ├── Vision-Language Model (Claude API)
    ├── Command Planning (Motor control sequences)
    └── Hardware Interface (receiver.py)
         └── Flask Server (receiver.py)
              └── Serial Communication → Arduino Firmware
                   └── Relay Control (Finger EMS)

stimGUIfingersHardMode.py (Manual Testing GUI)
    └── Direct Hardware Interface (PyQt5 + Serial)

Additional Demo: Obstacle Avoidance

Run a ball-detection demo that triggers avoidance responses:

python utils/ball_demo.py

Challenges we ran into

Troubleshooting

Arduino not detected:

• Check USB cable connection
• Verify correct board selection in Arduino IDE
• Confirm serial port settings (115200 baud)

No camera detected:

• Verify camera permissions (especially on macOS)

Missing API key:

• Ensure .env file exists in the root directory
• Verify ANTHROPIC_API_KEY is correctly set

Accomplishments that we're proud of

What we learned

What's next for Human Operator

Experimenting with EMS electrode size and placement to explore new applications on other parts of the body.

Inspiration

Inspired by research and systems from the Human Computer Integration Lab at UChicago on neuromuscular interfaces and electrode placement optimization:

• Full-Hand Electro Tactile Feedback without Obstructing Palmar Side of Hand
• Generative Muscle Stimulation: Providing Users with Physical Assistance by Constraining Multimodal-AI with Embodied Knowledge
• Increasing Electrical Muscle Stimulation’s Dexterity by means of Back of the Hand Actuation

Built With

• arduino
• claude
• opencv
• python
• tens-unit-ems