MediMetrics

Architecture
Hand Measurements
Face Measurements

Inspiration

When volunteering at a 3D printed prosthetics organization, one of our team members noticed a recurring problem: how hard it is to get accurate measurements from patients especially if they live halfway across the world in a rural/remote area. This often led to ill-fitting prosthetics, repeated shipments, and delayed treatment.

What it does

MediMetrics is a computer-vision-powered platform that transforms simple photos into precise, clinically relevant measurements. By combining AI landmark detection with scale calibration, it enables accurate remote assessments for health tracking, rehabilitation, and prosthetic design—no matter where the patient is.

For Patients:

Capture or upload photos from any smartphone
Instantly view measurements with clear visual overlays
Track growth, swelling, or recovery progress over time
Securely store data and control who has access to it

For Clinicians:

Request secure access to patient data
Review historical measurements with timestamps
Download CSV reports for EHR integration
Collect prosthetic or rehabilitation measurements without in-person visits

How we built it

Computer Vision & Scale Calibration:

Used OpenCV to detect ArUco markers in each uploaded image.
Computed a precise scale factor (mm per pixel) using the known size of the marker.
Corrected for camera angle and perspective to minimize distortion and produce reliable measurements.

Landmark Detection & Measurement:

Integrated MediaPipe for real-time hand and face landmark detection.
Computed distances between key landmark points (e.g., fingertips, wrist, jawline) and converted them into millimeter measurements using the previously established scale.
Designed a modular measurement function so we can easily add new types of measurements (e.g., foot length, arm span) in future updates.

Image Annotation & Output:

Overlaid measurement lines, labels, and markers directly on the image using OpenCV drawing functions.
Exported the annotated image as a base64-encoded PNG, alongside a structured JSON response containing all measurement data with exact coordinates and distances.

Computer Vision Custom API:

Built a FastAPI service with multiple endpoints (/measure-hand, /measure-face) to process different body parts.
Used Uvicorn as the ASGI server for high performance.
Exposed the API using Ngrok, since it is hosted locally on a computer.

Data & Authentication Layer:

Stored each measurement session in Firebase Firestore, including:
- Patient ID
- Measurement data
- Timestamp
- Link to annotated image in Firebase Storage
Implemented patient-controlled access permissions, so clinicians can request and patients can grant or revoke data access at any time.

Frontend Web App:

Developed a mobile-first React application Hosted on Vercel with a simple photo upload flow, real-time loading states, and instant results display.
Built a secure login and registration flow with Firebase Authentication, supporting both patients and clinicians.
Created a results view that shows the annotated image, numerical measurements, and options to save or discard the result.
Confirm or deny access request to results

Clinician Dashboard:

Secure login Firebase Authentication
Built a dedicated dashboard where healthcare professionals can:
- Browse patients they have access to
- View measurement history with timestamps and images
- Export data as CSV files for EHR integration
- Request access to patient's measurements by entering patient's email

Challenges we ran into

Initially, for the patient side of the app, we started by using React Native, despite having very little experience with it. Implementing the UI was relatively straightforward; however, React Native made authentication extremely complicated. After hours of failed attempts, we decided to transition to a React web app instead.

Additionally, our original plan for calibration was to use a simple piece of paper to measure distances and correct distortion caused by camera angles. This approach, however, proved to be quite tricky, and we struggled with consistent detection. After many failed attempts, we discovered Aruco markers, which turned out to be far more accurate and reliable.

Another major challenge we faced was deploying our computer vision API to Google Cloud. Everything worked perfectly on our local machines, but deploying it was a nightmare. No matter what we tried, we kept running into the same persistent error. After spending hours troubleshooting with no success, we ultimately decided to host the computer vision API locally and use Ngrok to make it accessible outside of local networks.

Accomplishments that we're proud of

We’re proud that, despite the technical hurdles, we built a complete end-to-end system—from image capture to delivering measurement results—that actually works. We overcame major challenges such as pivoting from React Native to React, integrating computer vision with FastAPI, and seamlessly connecting everything with Firebase. Most importantly, we created a tool that has the potential to genuinely expand access to healthcare, demonstrating that our idea is not only feasible but also impactful.

What we learned

Although we didn’t use React Native in our final project, our initial efforts to implement it taught us a great deal, especially since none of us had built a mobile app before. Working with so many different APIs and backend components pushed us to design a solid system architecture, giving us valuable experience in application design. Building our own computer vision API, hosting it locally, and deploying the frontend so that others could access it on their devices was a challenging but very rewarding experience.

What's next for MediMetrics

We plan to expand our measurement capabilities to cover the entire body, while also enabling doctors to annotate images and provide direct feedback through the app. In healthcare, this could support prosthetic fittings, orthopedic brace sizing, wound tracking, dermatology assessments, and even monitoring infant growth for early signs of malnutrition. Beyond medicine, MediMetrics has applications in other industries as well—for instance, nail technicians could use it to design custom press-on nails without requiring additional client visits, and jewelry designers could request precise finger measurements from clients remotely.