Anatomist

Web App UI
MediaPipe
Hand Anatomy
Hand Anatomy Explained

Inspiration

Studying anatomy is difficult without a clear sense of scale, depth, and spatial relationships. Textbooks flatten complex structures, and even 3D models feel disconnected from real-world context. We wanted a way to make anatomy instantly understandable by placing lifelike structures directly into a learner’s environment. That idea became the foundation for Anatomist.

What it does

Anatomist uses AR to overlay skeletal and muscular structures directly onto a user’s body in real time, showing exactly where each component sits beneath the skin. Each element includes clear, position-accurate labels that identify individual bones, muscles, and regions. A built-in AI explainer activates through a simple hand gesture, providing detailed, on-demand explanations of any highlighted part.

How we built it

We built Anatomist with Next.js and React Three Fiber for the AR interface, using Three.js and three-vrm to render and align anatomy models. MediaPipe Pose provides real-time body tracking, supported by camera_utils and drawing_utils, while Kalidokit stabilizes pose data for accurate overlays. The interface uses Radix UI, lucide-react, and TailwindCSS for a clean, responsive layout. This stack lets us map skeletal and muscle layers onto the user and trigger AI explanations through simple gestures.

Challenges we ran into

The biggest challenge was accurately positioning the skeletal and muscle overlays in Three.js so they aligned perfectly with the real-time body tracking from MediaPipe Pose. Translating 2D keypoints into a 3D space and keeping the overlays stable during movement required careful calibration and iterative adjustments. Additionally, we had to individually create our own Blender models for each bone group. Models on the Internet did not feature individual selection of each bone, so we had to create our own groups for each.

Accomplishments that we're proud of

We created a fully interactive AR anatomy experience that overlays skeletal and muscle structures directly onto a user’s body, complete with clear labels and a gesture-activated AI explainer. Beyond mastering the technical challenge of real-time 3D alignment with MediaPipe tracking, we’re proud of the broader potential: Anatomist can transform how people learn human anatomy, making education more immersive, accessible, and intuitive. This technology could benefit medical students, trainers, and hobbyists alike, bridging the gap between theory and real-world understanding and paving the way for AR-driven learning in other complex subjects.

What we learned

We learned a lot about building immersive AR experiences, including how to map real-time body tracking from MediaPipe onto 3D models in Three.js. We discovered the challenges of maintaining stable overlays in a moving environment and how to design intuitive interactions, like hand gestures for AI explanations. This project gave us hands-on experience with the unique constraints and possibilities of AR for education.

What's next for Anatomist

Next, we plan to expand Anatomist with more body systems and improve overlay accuracy and responsiveness. By scaling and adding muscle groups and underlying organ systems, students and educators will be able to see in real time how each system works, including the contraction of their own muscles. Additionally, we plan to include more sensors and support for DICOM images so that medical researchers can use our app to diagnose and treat possible problems such as bone fractures and immunodeficiencies. Beyond technical updates, we aim to explore its impact on education—bringing immersive, hands-on anatomy learning to classrooms, labs, and remote learners. By making complex structures tangible in AR, Anatomist could change how students, educators, and medical trainees understand the human body.