PhysioLens

Inspiration

Two months ago, our teammate Sherwin underwent surgery. His recovery required weekly physiotherapy sessions where he'd receive new exercises to perform at home. The problem? Small mistakes with catastrophic consequences.

Picture this: Sherwin performs his bicep curls but doesn't realize his elbow is drifting forward. His shoulder rotates slightly during lateral raises. His back arches during shoulder presses. These aren't major failures, they're subtle form deviations that feel "close enough."

But in physical therapy, "close enough" isn't good enough.

The reality is brutal:

• Recovery times extended by weeks due to incorrect form
• Billions of dollars wasted on preventable extended therapy
• No feedback between weekly PT sessions—you're flying blind
• Patients don't know they're doing exercises wrong until their next appointment
• Thousands of patients make similar mistakes every single day

I realized that traditional PT gives you exercises on paper, sends you home, and hopes you do them correctly. There's zero real-time feedback, zero performance tracking, and zero accountability between sessions.

We needed something that meets patients where they are—at home—and provides the professional-grade coaching they'd get in a clinic. Something that uses AI to analyze every rep, catches mistakes before they become habits, and gives doctors actual data instead of patient self-reports.

PhysioLens was born from this frustration. It's the AI physical therapist I wish Sherwin had during his recovery.

What it does

PhysioLens is an AI-powered physical therapy platform that transforms your home into a professional rehabilitation clinic. Just turn on your camera, and watch the magic happen:

🎯 For Doctors: AI-Powered Exercise Creation

• Type a description: "Tricep extensions for post-surgical shoulder rehabilitation"
• Claude AI instantly generates:
  • Camera positioning requirements
  • Precise angle thresholds for rep counting
  • Step-by-step patient instructions
  • Automated form checks
  • PubMed research references supporting the exercise protocol
• Production-ready exercise configuration in 30 seconds to 5 minutes
• Built-in library: 5 professionally configured exercises
• Unlimited custom AI-generated exercises

🏃‍♂️ For Patients: Real-Time AI Coaching

• Dual video feed: Raw camera + AI skeleton overlay
• 10-second countdown with on-screen positioning guidance
• Automatic rep counting using biomechanical angle analysis
• Instant form warnings: "Elbow drifting - keep it stable!", "Using momentum - slow down!"
• Real-time angle display with target ranges
• Phase tracking: "Down" → "Up" with progress bar
• Target completion popup when reps reached

🤖 Meeting Mode: Voice-Powered Clinical Assistant

• Continuous voice transcription during exercise sessions
• Emergency detection: "My knee hurts really bad" → Instant alert with urgency score
• Meeting scheduling: "Let's schedule a follow-up next Tuesday at 3 PM" → Auto-captured and calendared
• AI-generated clinical summaries after every session:
  • Chief complaint extraction
  • Performance + conversation insights
  • Patient mood assessment
  • Compliance level tracking
  • Follow-up recommendations

Advanced Session Analysis

AI Performance Scoring (0-100):

• Weighted algorithm considering reps completed, form issues, and severity
• Scoring philosophy: 80-85 is excellent (perfect form is rare)

Biomechanical Issue Detection (9 Categories):

• Tremor/Instability, Compensation Patterns, Speed Variations
• Range of Motion Issues, Balance Problems, Form Deviations
• Asymmetry Detection, Fatigue Signs, and more
• Click any issue → jumps to exact timestamp in video

Clinical Research Integration:

• PubMed API: Automatic research references
• BrightData Web Scraping: NICE, NHS, CSP clinical guidelines
• Evidence-based recommendations with clickable citations

Clinical Notes Dashboard

• AI-generated clinical summaries for every session
• Scheduled meetings from voice transcriptions
• Performance metrics tracking
• Complete patient records

How we built it

PhysioLens is a sophisticated full-stack application combining AI, computer vision, and clinical research:

Frontend

• React 18 with Vite for lightning-fast development
• MediaPipe Pose for 33-point skeletal tracking
• React Router v6 for client-side routing
• Custom utilities for angle calculations and rep counting
• Real-time pose detection at 30 FPS with <50ms latency

Backend

• FastAPI (Python 3.8+) for high-performance async API
• Pydantic v2 for request/response validation
• In-memory storage for MVP (production-ready for database migration)
• 12 RESTful API endpoints for exercise management, session recording, and research

AI Engine

• Claude Sonnet 4 for exercise generation, performance analysis, and clinical summaries
• Structured JSON output with prompt engineering
• Context window management for efficient processing
• Multi-model approach for different use cases

Research Integration

• PubMed API for medical research papers
• BrightData Web MCP with Claude Agent SDK
• Agentic scraping of NICE, NHS, and CSP websites
• Structured extraction with proper citations

Voice Processing

• Web Speech API for browser-native transcription
• Real-time processing with auto-restart logic
• Emergency and meeting keyword detection
• Client-side processing for privacy

Video Recording System

• MediaRecorder API for session recording
• Pose data logging (33 landmarks × 3D coordinates)
• LocalStorage for session persistence
• Blob creation and URL generation

Rep Counting Algorithm

• Generic angle-based counter with phase state machine
• 15° hysteresis prevents false triggers
• 5-frame exponential moving average for smoothing
• 95%+ accuracy compared to manual counting

Advanced Analysis Engine

• SessionAnalyzer with 9 detection modules
• Processes sessions 5-10x faster than real-time
• Smart deduplication prevents spam
• Issue confidence scoring and severity classification

Key Technologies

{
  "@mediapipe/pose": "^0.5.1675469404",
  "fastapi": ">=0.115.0",
  "anthropic": "latest",
  "claude-agent-sdk": "latest",
  "react": "^18.2.0"
}

Challenges we ran into

🌐 BrightData Integration Evolution

Initial Approach: Started with BrightData's Browser API for static site structures, which limited website coverage and response quality.

The Breakthrough: During a sponsor panel session, BrightData recommended switching to their agentic framework with MCP tools integrated.

What changed:

• Agentic search with Claude AI intelligently queries multiple sources
• MCP tools (search_engine, scrape, grep) for advanced extraction
• Pro Mode unlocked full tool access
• Context-aware medical terminology understanding
• Result: 9-12 high-quality clinical references per exercise vs 0-2 broken snippets before

⏱️ 10-Second Countdown Implementation

MediaPipe onPoseResults callback captured stale state due to closures. React state updates didn't re-register the callback.

Solution: Used useRef instead of useState for mutable values that callbacks need to access.

🎥 MediaRecorder State Management

Recording would sometimes stop unexpectedly or duplicate chunks. Solved with dual state tracking and safe start/stop guards.

🧹 Cleanup & Memory Leaks

Memory usage climbing and camera not releasing when navigating away. Implemented comprehensive cleanup function with double-cleanup prevention.

🎯 False Positive Form Warnings

Users got spammed with warnings. Drastically increased thresholds (5x for tremor, 33% for lateral lean) and added better deduplication.

⚡ AI API Rate Limits

Hit Claude API rate limits during testing. Added random delays and planned request queue with exponential backoff for production.

🕐 Time Pressure

Built the entire system in under 36 hours. Had to cut features like user authentication and multi-patient support to focus on core MVP.

The lesson? Ship a great core experience beats shipping 10 half-broken features.

Accomplishments that we're proud of

✅ We Actually Finished (And Shipped Early!)

• Completed 6 hours before the deadline
• All core features working end-to-end
• Zero critical bugs in final build
• Professional-grade polish

🎯 Real Impact Potential

The numbers that matter:

• Average PT session: $150-200
• Typical patient needs: 12-24 sessions ($1,800-4,800 total)
• If PhysioLens reduces sessions by 20%: $360-960 saved per patient
• Over 10 million Americans undergo PT annually
• Billions of dollars wasted on preventable extended therapy

If PhysioLens helps just 1,000 patients:

• $360,000-960,000 in healthcare savings
• 20,000-60,000 hours of patient time saved
• Countless improved quality-of-life outcomes

🤖 AI That Actually Works

• Exercise generation: ~95% success rate, 30 seconds to 5 minutes
• Performance analysis: Processes 5,000+ pose frames per session
• Clinical summaries: Structured medical note format
• Emergency detection: Zero false negatives, ~5% false positive rate

🏗️ Engineering Excellence

• Smart rep counting with 95%+ accuracy
• Advanced analysis 5-10x faster than real-time
• Beautiful UX that doesn't suck
• Full-stack mastery across 8+ technologies

💡 We Learned From Our Mistakes

The BrightData pivot showed the power of asking for sponsor help early. Went from failing static scraping to getting 10x better results with agentic framework.

🏆 Built Something We're Proud to Show

Demoed to PT patients, physical therapists, and hackathon judges—all saw real value. This could be a real business.

What we learned

🎓 From Our Sponsors

Anthropic & Claude AI:

• Prompt engineering is everything—specificity drives quality
• Context window management for efficient processing
• Function calling for structured JSON output
• Cost optimization strategies ($0.05-0.15 per session)
• Claude's medical knowledge is genuinely impressive

BrightData & Web Scraping:

• Agentic scraping > Traditional scraping
• MCP gives Claude powerful extraction tools
• Pro Mode unlocks full capabilities
• 5-minute sponsor conversation > hours of documentation

OpenEvidence:

• Healthcare data must be evidence-based, not anecdotal
• Every recommendation needs research citations
• Higher bar for medical tools than consumer apps

Zingage:

• Understanding the patient journey is critical
• Healthcare products must be empathetic, not just functional
• Focus on emotional states patients experience

Stanford TreeHacks:

• Hackathons are innovation accelerators
• Constraints drive creativity
• Community makes the difference
• Impact over perfection

🛠️ Technical Skills Leveled Up

• Computer vision (MediaPipe, angle math, skeleton rendering)
• Voice processing (Web Speech API, continuous transcription)
• AI engineering (prompt engineering, RAG, cost optimization)
• Web scraping (agentic scraping, MCP tools)
• State management (useRef vs useState, cleanup functions)
• Healthcare domain (PT workflows, biomechanics, clinical notes)

💡 Product Philosophy

The biggest learning: Technology is easy. Impact is hard.

Start with the problem, not the tech. Talk to users. Measure what matters. Build for outcomes.

What's next for PhysioLens

🚀 Immediate Optimization

• User testing with 10-20 real PT patients
• Production infrastructure (PostgreSQL, authentication, cloud deployment)
• Security hardening (HIPAA compliance considerations)
• Cost optimization for scale

📱 Mobile App

• Native iOS/Android apps (80% of PT patients prefer mobile)
• Better camera angles and movement tracking
• Offline mode for exercises without internet

🌐 Multi-Provider Platform

• Support for multiple doctors and clinics
• Patient portal for progress tracking
• Doctor dashboard for patient management
• Insurance integration for reimbursement

🔬 Advanced Features

• Exercise progression algorithms
• Predictive analytics for recovery timelines
• Integration with wearables (Apple Watch, Fitbit)
• Telemedicine video consultations

💼 Go-to-Market Strategy

• Pilot program with 3-5 local PT clinics
• Pricing model: $29/month per patient or clinic licensing
• Insurance partnerships for coverage
• Clinical trials for FDA clearance (if needed)

The vision: Make professional-grade physical therapy accessible to everyone, everywhere. PhysioLens could save billions in healthcare costs and help millions recover faster.

Acknowledgments

• Anthropic - Claude AI for exercise generation and analysis
• Google MediaPipe - Pose detection technology
• PubMed/NCBI - Medical research database
• BrightData - Web scraping infrastructure
• OpenEvidence - For empowering us to create innovative healthcare solutions
• Zingage - For supporting our journey in building impactful health technology products
• Stanford TreeHacks - For providing the opportunity to participate in this incredible hackathon and bring PhysioLens to life

Built With

• brightdata-web-mcp
• claud-sonnet-4
• fastapi
• mediapipe
• react
• web-speech-api