MimeCraft

Landing page
5 angles captured on a slice of pizza. 3D model created

Inspiration

We were inspired by how powerful 3D modeling and CAD tools are, but also how difficult they can be for beginners to learn. At the same time, with the rise of 3D printing, anyone can bring their ideas into the real world, that is... if they know how to model them. We wanted to lower that barrier and create something intuitive and creative: a tool that lets people shape and build ideas as naturally as using their hands. Our goal was to make 3D creation feel less like technical work and more like sculpting with imagination.

What it does

MimeCraft turns the world into your creative playground. Users can take an image or video (or upload one) and instantly convert it into a fully editable 3D model. From there, they can reshape, refine, and interact with the model in real time using natural hand gestures and voice input.

Instead of learning complex tools, users can: Pinch to pull and sculpt the model Point to create indents and details Use hand gestures to zoom and explore Modify their creation intuitively, as if molding clay MimeCraft makes it possible to transform imagination into interactive 3D creations. Anytime, anywhere.

How we built it

We combined multiple technologies to create a seamless pipeline from image to interaction: Gemini API is used to analyze input images and generate descriptive text of the object That text is passed into Meshy, which generates a 3D model from the descriptive text A real-time hand tracking system using MediaPipe Hands detects gestures The 3D environment is rendered and manipulated using Three.js Together, these components allow users to go from a simple image to an interactive 3D model they can sculpt with their hands.

Challenges we ran into

This project came with a lot of challenges. Integrating multiple systems, AI image processing, 3D generation, and real-time hand tracking was complex and required careful coordination. One of the biggest challenges was making gesture recognition reliable and responsive. Distinguishing between similar gestures (like pinch vs. fist) and ensuring smooth, continuous interaction with the 3D model took significant debugging and refinement.

Accomplishments that we're proud of

We’re proud that we were able to bring all these components together into a working system. Successfully converting images into 3D models Implementing real-time hand gesture controls Creating an interactive sculpting experience in the browser Most importantly, we built something that actually works and demonstrates a new, more intuitive way to interact with 3D content.

What we learned

We learned the importance of persistence and iteration. Many parts of this project didn’t work at first, and we had to continuously debug, refine, and rethink our approach. We also learned how powerful it is to combine different technologies, AI, computer vision, and 3D graphics to create something entirely new.

What's next for Mime Craft:

Next, we want to expand MimeCraft beyond single objects. We aim to allow users to scan picture of entire environments (e.g., rooms, landscapes) and convert them into fully interactive 3D spaces. Users would be able to explore, edit, and redesign these environments in real time.

This could be useful for: Game map design (for exploration games) Interior design and planning Virtual exploration and visualization

Our goal is to help people not just imagine ideas but step inside them and shape them.