Hand shadow puppet recognition system with MediaPipe integration and TouchDesigner streaming

Shadow-Vision transforms hand shadow puppet gestures into real-time data streams for interactive media installations. Built around Google's MediaPipe framework, the system achieves 91.9% recognition accuracy while streaming both gesture classifications and raw hand landmark data to TouchDesigner for immediate creative application.

• 21 landmark tracking per hand with sub-pixel accuracy
• Dual-hand simultaneous detection for complex gesture combinations
• 91.9% classification accuracy across 8 distinct shadow puppet animals
• Real-time performance at 30+ FPS with minimal latency

• Complete OSC protocol streaming gesture data and raw landmarks
• 63 coordinates per hand delivered at full framerate
• Configurable data formats supporting both individual coordinates and array streams
• Live camera feed integration with gesture overlay rendering

• Docker containerization solving MediaPipe version compatibility
• Environment isolation supporting Python 3.10-3.12 while maintaining system stability
• Automated setup scripts for both development and production environments
• Comprehensive testing suite validating accuracy and communication protocols

The system recognizes eight distinct shadow puppet gestures with the following accuracy metrics:

Gesture Classification

/shadow_puppet/gesture        - Combined results (e.g., "L:bird+R:cat")
/shadow_puppet/left_index     - Animal class index for left hand
/shadow_puppet/right_index    - Animal class index for right hand
/shadow_puppet/confidence     - Detection confidence (0.0-1.0)

Raw Hand Landmarks

/landmarks/left/0/x           - Wrist position (normalized)
/landmarks/left/thumb_tip/x   - Named landmark access
/landmarks/left/array         - Complete 63-element coordinate array

Advanced Hand Metrics

/landmarks/left/finger_0/angle     - Finger bend angles
/landmarks/left/hand_span_x        - Hand dimensions
/landmarks/left/palm_center        - Palm positioning

git clone https://github.com/pablomoli/Shadow-Vision.git
cd Shadow-Vision

Linux:

# Full camera access and GUI support
docker-compose -f docker-compose.touchdesigner.yml up --build mediapipe-bridge

Windows:

# Platform-optimized configuration
docker-compose -f docker-compose.windows.yml up --build mediapipe-bridge

macOS:

# Platform-optimized configuration
docker-compose -f docker-compose.macos.yml up --build mediapipe-bridge

Platform-Specific Requirements:

• Windows: Ensure Docker Desktop has camera access permissions enabled
• macOS: Enable camera access in Docker Desktop → Preferences → Resources
• Linux: Native /dev/video0 device mapping (no additional setup required)

Automated Setup (All Platforms)

# Cross-platform automated setup
python setup_mediapipe_env.py

Manual Setup

Linux/macOS:

python3.11 -m venv mediapipe_env
source mediapipe_env/bin/activate
pip install -r requirements-mediapipe.txt

Windows:

py -3.11 -m venv mediapipe_env
mediapipe_env\Scripts\activate.bat
pip install -r requirements-mediapipe.txt

Alternative Windows (PowerShell):

python -m venv mediapipe_env
mediapipe_env\Scripts\Activate.ps1
pip install -r requirements-mediapipe.txt

1. Add OSC In CHOP component
2. Configure network settings:
   • Port: 7000
   • Address: 127.0.0.1
   • Auto Update: Enabled

Camera Input → MediaPipe Processing → Gesture Classification → OSC Streaming → TouchDesigner
     ↓              ↓                        ↓                    ↓              ↓
Hand Detection  Landmark Extraction    ML Inference         Data Formatting   Creative Output
21 points/hand    63 coordinates         91.9% accuracy      60+ msg/sec       Real-time render

The system operates through distinct processing stages:

Computer Vision Pipeline

• MediaPipe hand detection with confidence thresholding
• Real-time landmark extraction at camera framerate
• Coordinate normalization and stability filtering

Machine Learning Inference

• Trained ensemble model using MediaPipe features
• Gesture classification with confidence scoring
• Temporal smoothing for stable output

Communication Layer

• OSC message formatting for TouchDesigner compatibility
• Configurable data streaming options
• Error handling and connection recovery

Shadow-Vision/
├── Core Recognition Engine
│   ├── enhanced_mediapipe_touchdesigner_bridge.py    # Primary TouchDesigner interface
│   ├── mediapipe_touchdesigner_bridge.py             # Simplified gesture-only bridge
│   ├── live_two_hand_demo.py                         # Standalone demonstration
│   └── train_mediapipe_model.py                      # Model training pipeline
│
├── Backend Processing
│   ├── backend/data/mediapipe_extractor_real.py      # Real MediaPipe landmark extraction
│   ├── backend/data/advanced_feature_extractor.py    # Feature engineering pipeline
│   ├── backend/models/gesture_classifier.py          # ML model architecture
│   └── models/mediapipe_*.joblib                     # Trained model files
│
├── Deployment Infrastructure
│   ├── Dockerfile.bridge                             # MediaPipe container configuration
│   ├── docker-compose.touchdesigner.yml              # TouchDesigner integration setup
│   ├── requirements-mediapipe.txt                    # Version-locked dependencies
│   └── setup_mediapipe_env.py                       # Automated environment configuration
│
├── Documentation & Testing
│   ├── TOUCHDESIGNER_OSC_REFERENCE.md               # Complete OSC message documentation
│   ├── DEPLOYMENT_GUIDE.md                          # Production deployment guide
│   ├── test_mediapipe_accuracy.py                   # Accuracy validation suite
│   └── test_osc_bridge.py                          # Communication testing
│
└── Sample Data & Configuration
    ├── data/mediapipe/                              # Processed training dataset
    ├── config/gesture_mappings.json                 # Gesture configuration
    └── validate_setup.py                           # System validation

• Overall accuracy: 91.9% on validation dataset
• Processing latency: Sub-30ms per frame
• Detection range: 0.5-3.0 meters from camera
• Lighting tolerance: Indoor to bright outdoor conditions

• CPU utilization: 15-25% on modern processors
• Memory footprint: ~200MB per bridge instance
• Network throughput: 60+ OSC messages per second
• Camera resolution: Supports 640x480 to 1920x1080

• Message delivery: Zero-copy OSC streaming
• Coordinate precision: Normalized floating-point values
• Update frequency: Matches camera framerate
• Data formats: Individual coordinates, arrays, or hybrid streaming

The enhanced bridge supports multiple landmark data formats:

• Individual coordinates: Separate OSC messages per landmark point
• Array format: Complete hand data in single 63-element array
• Named landmarks: Semantic addressing (wrist, thumb_tip, etc.)
• Derived features: Finger angles, hand span, palm center calculations

Built-in temporal filtering prevents false detections:

• Confidence thresholding: Configurable detection sensitivity
• Stability duration: Gesture confirmation timing
• Transition smoothing: Gradual transitions between gesture states

• Automatic recovery: Connection and camera failure handling
• Performance monitoring: Real-time FPS and accuracy tracking
• Debug visualization: Optional camera feed with gesture overlays
• Logging integration: Comprehensive error reporting and diagnostics

# Validate model performance against test dataset
python test_mediapipe_accuracy.py

# Test real-time recognition stability
python test_two_hand_detection.py

# Verify OSC message delivery
python simple_osc_test.py

# Test complete TouchDesigner integration
python test_enhanced_bridge.py

# Comprehensive setup verification
python validate_setup.py

# Docker deployment testing
python test_docker_setup.py

# Verify Python version compatibility (3.10-3.12)
python --version

# Use Docker for automatic dependency resolution
docker-compose -f docker-compose.touchdesigner.yml up mediapipe-bridge

All Platforms:

# Test camera availability
python -c "import cv2; print('Camera:', cv2.VideoCapture(0).isOpened())"

Linux:

# Add user to video group for camera permissions
sudo usermod -a -G video $USER
# Then logout and login again

Windows:

• Check camera permissions in Windows Settings → Privacy → Camera
• Ensure Docker Desktop has camera access enabled
• Try different camera indices (0, 1, 2) if default doesn't work

macOS:

• Enable camera access in System Preferences → Security & Privacy → Camera
• Grant Docker Desktop camera permissions
• Restart Docker Desktop after enabling permissions

Docker-Specific Issues:

# Windows/macOS: Use platform-specific compose files
docker-compose -f docker-compose.windows.yml up mediapipe-bridge  # Windows
docker-compose -f docker-compose.macos.yml up mediapipe-bridge    # macOS
docker-compose -f docker-compose.touchdesigner.yml up mediapipe-bridge  # Linux

# Verify OSC communication
python simple_osc_test.py

# Check TouchDesigner OSC In CHOP configuration:
# Port: 7000, IP: 127.0.0.1, Auto Update: On

1. Collect training data using MediaPipe landmark extraction
2. Retrain classification model with expanded dataset
3. Update gesture mapping configuration
4. Validate performance with accuracy testing suite

• Gesture switching: Use classification indices for model/scene selection
• Hand positioning: Apply landmark coordinates to 3D object transforms
• Finger control: Utilize individual finger angles for detailed animation
• Hybrid interaction: Combine gesture recognition with positional tracking

• Reduce landmark streaming: Disable coordinate streaming for gesture-only applications
• Adjust confidence thresholds: Balance detection sensitivity with false positive rates
• Camera resolution scaling: Lower resolution for improved performance on limited hardware

GitHub Actions continuously validates cross-platform compatibility:

• Linux: ✅ Virtual environments, Docker syntax, Python scripts
• Windows: ✅ Virtual environments, Docker syntax, Windows-specific commands
• macOS: ✅ Virtual environments, Docker syntax, macOS-specific commands

While automated tests validate syntax and basic functionality, real-world testing with cameras and TouchDesigner requires community validation:

• Camera access testing: Docker camera permissions on Windows/macOS
• TouchDesigner integration: OSC communication across platforms
• Performance validation: 30+ FPS gesture recognition on various hardware

Help us test! See PLATFORM_TESTING.md for detailed testing instructions.

The project welcomes contributions in several areas:

• Gesture expansion: Additional shadow puppet animals or hand poses
• Performance optimization: Algorithm improvements and efficiency gains
• TouchDesigner examples: Sample projects demonstrating integration patterns
• Cross-platform testing: Validation on different operating systems and hardware

• Complete OSC Reference
• Deployment Guide
• Integration Summary

• Issue Tracking
• Development Discussions

Built for ShellHacks 2025 | Real-time Interactive Media Applications

Transforming hand gestures into immersive digital experiences through advanced computer vision and real-time data streaming.